Mouad Imam
All articles

12 min read

Sidekiq to Solid Queue in a nine-year-old Rails app: a simulated walkthrough

A simulated migration of a nine-year-old Rails monolith from Sidekiq to Solid Queue: what maps, what has no equivalent, and how to cut over safely.

To keep this concrete without exposing anyone's code, this article works through a simulated codebase: a nine-year-old Rails monolith whose background work grew up around Sidekiq. It is a composite I made up, not a report of a real migration, and every count below belongs to the simulation. I have worked on Sidekiq jobs in production Rails applications, so the setup will look familiar.

The Rails 8 release notes say Solid Queue replaces the need for Redis and for a job framework such as Sidekiq1. For a new app that is easy. For an old one the question is how much of the app quietly depends on Sidekiq. In this app the answer is to move most jobs and leave a few.

The simulated app

  • 180 job classes. 70 are native workers (include Sidekiq::Job, some older ones Sidekiq::Worker) called with perform_async and perform_in. 110 are Active Job.
  • config/sidekiq.yml with weighted queues.
  • sidekiq_options with custom retry counts, dead: false and backtrace: true, plus 9 sidekiq_retries_exhausted hooks that notify someone.
  • Server middleware that switches tenant, client middleware that adds a request id, and one that records metrics.
  • 14 recurring jobs through sidekiq-cron.
  • 6 jobs guarded by sidekiq-unique-jobs locks.
  • 2 multi-step exports built on Sidekiq Pro batches with callbacks.
  • 1 partner API call behind an Enterprise rate limiter.
  • Invoice reminders scheduled weeks ahead, and a dead set people retry by hand.
  • An admin screen and a health check that read Sidekiq::Stats and queue latency. The Web UI is mounted behind admin auth.
  • Some jobs enqueued inside database transactions.
  • An error tracker, and a deploy script that sends TSTP, then TERM.

Discovery

Inventory before you design. These greps are crude and enough:

grep -rnE "include Sidekiq::(Job|Worker)" app lib | wc -l
grep -rnE "perform_(async|in|at)\b" app lib | wc -l
grep -rnE "sidekiq_options|sidekiq_retries_exhausted|sidekiq_retry_in" app lib
grep -rnE "client_middleware|server_middleware|death_handlers" config
grep -rnE "unique_for|lock:|Sidekiq::(Limiter|Batch|Cron)" app lib config
grep -rnE "Sidekiq::(Queue|ScheduledSet|RetrySet|DeadSet|Stats|ProcessSet)|Sidekiq\.redis" app lib config
grep -nE "sidekiq" Gemfile Procfile* config/deploy* bin/* 2>/dev/null
cat config/sidekiq.yml

Read the Redis hits by hand. Anything that touches Redis or the Sidekiq API directly will not survive the move.

What maps and what does not

Sidekiq todaySolid QueueVerdict
Weighted queues [critical, 4]Queues polled in the order listed in queue.yml. Per-job priority, lower runs first, queue order wins2Behaviour change: weights become strict order
sidekiq_options retry:, dead:retry_on, discard_on. Unretried failures go to failed executions3Rewrite per class
sidekiq_retries_exhausted, death handlersBlock form of retry_on4Rewrite
Client and server middlewareActive Job callbacks such as before_enqueue and around_perform3Rewrite
sidekiq-cron, Enterprise periodic jobsconfig/recurring.yml2Port
sidekiq-unique-jobs, Enterprise unique jobslimits_concurrency2Partial
Enterprise rate limitingNo documented equivalentStay on Sidekiq
Pro batchesSolidQueue::Batch, since 1.7.05Rewrite and test, or stay
perform_in, perform_atset(wait:).perform_later3Port
Web UIMission Control Jobs6Replace
Sidekiq::Stats, Queue in admin screensNoneRewrite
TSTP then TERM7TERM or INT, then QUIT after shutdown_timeout2New deploy script

Where there is no clean equivalent

Rate limiting is the plainest gap. The Enterprise limiter offers concurrent, bucket, window, leaky bucket and points types8. The Solid Queue README documents limits_concurrency, which caps jobs running at once for a key2. That resembles the concurrent type. I found nothing for the others. The partner API job stays on Sidekiq.

Batches are harder. Sidekiq's wiki says batches don't work well with Active Job because an Active Job retry looks like a success to Sidekiq, and recommends native jobs9. So the two exports can't move casually even to Sidekiq-backed Active Job. Solid Queue added batches in 1.7.05, with on_finish, on_success and on_failure callbacks that fire after retries are done. The README also says installations that predate batches need a migration, and that the batch tables join the base schema in 2.02. I would rewrite one export on it, test it with forced failures, and decide then. Until that passes, the exports stay.

Uniqueness maps partly. Per the README, limits_concurrency claims a semaphore at enqueue, releases it when the job finishes, and with on_conflict: :discard drops jobs enqueued while it is closed2. For jobs enqueued to run now, that reads like an until_executed lock10, with duration (default 3 minutes) as the expiry2. Scheduled jobs are limited when due, not when enqueued2, which differs from locks held across the delay. Check all 6 jobs against that.

Weighted queues become ordered ones. Decide whether starving maintenance behind critical is acceptable.

Order of work

1. Run Solid Queue beside Sidekiq

Follow the Solid Queue README: bundle add solid_queue, then bin/rails solid_queue:install. It generates config/queue.yml, config/recurring.yml, db/queue_schema.rb and bin/jobs2. Rails 7.1 and Ruby 3.2 are the minimum2; a nine-year-old app may need the upgrade first, which is its own project.

Use the separate queue database the README recommends, with a queue entry in config/database.yml and migrations_paths: db/queue_migrate2. Leave the global adapter on Sidekiq. Start bin/jobs, or the Puma plugin on a single server2, and route nothing yet. A queue file that preserves today's order:

production:
  workers:
    - queues: [critical, default, reports, maintenance]
      threads: 3
      processes: 1

2. Make arguments safe

Sidekiq stores arguments as JSON. The wiki says they must be simple JSON types and not to pass symbols11. Active Job's guide lists more types as supported, among them Symbol, Date, Time and Hash with string or symbol keys, plus GlobalID for Active Record objects3. Old native workers that take ids and string-keyed hashes keep working. Look for code that quietly relied on JSON turning symbol keys into strings. Active Job raises ActiveJob::DeserializationError when a record has been deleted before the job runs3; decide per job whether to discard.

3. Convert native workers, still on Sidekiq

Convert a worker to an Active Job class first and keep it on the Sidekiq adapter. That separates the code change from the backend change. Retries need care here. Sidekiq's Active Job page says that once retry_on gives up, Active Job hands the job back to Sidekiq's own retries12. On Solid Queue that second layer is gone. The Active Job API says an exhausted job without a block re-raises4, and the Solid Queue README says a failed job stays as a failed execution until someone retries or discards it2. Sidekiq's default is 25 retries over about 20 days13. Write the policy down per class:

class InvoiceReminderJob < ApplicationJob
  queue_as :default
  retry_on Net::OpenTimeout, wait: :polynomially_longer, attempts: 8
  discard_on ActiveJob::DeserializationError

  def perform(invoice) = invoice.send_reminder!
end

retry_on takes a block that receives the job and the error4, which replaces sidekiq_retries_exhausted hooks.

4. Replace middleware with callbacks

The callbacks section of the guide lists before_enqueue, around_enqueue, around_perform and the rest3. Put them in ApplicationJob:

class ApplicationJob < ActiveJob::Base
  around_perform :with_tenant

  private

  def with_tenant
    Tenant.switch(arguments.first.tenant_id) { yield }
  end
end

The tenant has to travel with the job. Pass it as an argument, or register a custom serializer, which the guide documents3. Note that perform_all_later skips around_enqueue3. The metrics middleware is the easiest to port. The tenant one deserves tests first.

5. Move low-risk classes

The README documents the incremental path: keep config.active_job.queue_adapter on the old backend and set the adapter inside each job you move2.

class ReportCleanupJob < ApplicationJob
  self.queue_adapter = :solid_queue
end

Start with idempotent jobs nobody pages on. Fail one on purpose and look at it in Mission Control. Sidekiq makes no exactly-once promise11, so these jobs should already be idempotent. Check that they are.

6. Move recurring jobs

Each sidekiq-cron entry becomes a recurring.yml entry with a class or command, a schedule, and optionally args, queue and priority2. Schedules are parsed by Fugit and default to config.time_zone2. Run one scheduler at a time: delete the sidekiq-cron entry in the same deploy that adds the new one, or the job runs twice. If you use sidekiq-scheduler with sub-minute cron or every entries14, check them against Fugit before moving them.

7. Handle enqueue inside transactions

Sidekiq pushes to Redis immediately, outside your database transaction. The default separate queue database has the same property. The README warns that putting jobs in your application database is a "powerful yet sharp tool"2. Rails 7.2 added enqueue_after_transaction_commit, which defers enqueuing until the transaction commits. In Rails 8.0 and 8.1 it is a per-job class setting and off by default; the global config was deprecated in 8.0. On 7.2 the global default let the adapter decide15, so check which behaviour the app had under Sidekiq. Enable it where jobs read rows the transaction writes, and test each one.

Operations as the README describes them

  • Threads: keep a worker's threads at or below the queue database pool size minus 22.
  • Database: the README says Solid Queue is designed for the highest throughput on MySQL 8+, MariaDB 10.6+ or PostgreSQL 9.5+. For older versions set use_skip_locked to false. On MySQL and MariaDB it recommends READ COMMITTED2.
  • Retention: finished jobs are preserved for a day by default, and a recurring task clears them2.
  • Deploys: TERM or INT starts a graceful stop, and the supervisor sends QUIT after shutdown_timeout, which defaults to 5 seconds against Sidekiq's 2527. I found no TSTP-style quiet signal in the README, so the deploy script changes. A process that dies without finishing leaves its claimed jobs to be marked failed with ProcessPrunedError once its heartbeat expires, by default after 5 minutes2. They are not retried for you.

Drain before removing Sidekiq

Stop enqueueing to Sidekiq class by class, then read the API16:

require "sidekiq/api"

Sidekiq::Queue.new("default").size # repeat per queue
Sidekiq::ScheduledSet.new.map(&:display_class).tally
Sidekiq::RetrySet.new.size
Sidekiq::DeadSet.new.size
Sidekiq::ProcessSet.new.each { |p| puts p["busy"] }

Stats#enqueued excludes retries and scheduled jobs16, so don't trust it alone. Set iteration is best effort while the sets change16.

The scheduled set is the trap. Reminders set for next month sit in Redis. The API wiki page does not document their run time, but each entry has an at method in the source, next to display_class and args17. Either keep a small Sidekiq process running until the set empties, or re-enqueue each entry in Solid Queue for entry.at and delete the original, accepting a double-run risk for non-idempotent ones.

Retry what is worth retrying with select { ... }.map(&:retry)16 while Sidekiq is still up. For the rest, delete on purpose, not by removing Redis.

Monitoring afterwards

Mount Mission Control Jobs. HTTP Basic auth is on by default and the UI stays closed until you set credentials. It needs Solid Queue 1.0.1 or later and supports one Solid Queue configuration6. Rewrite the admin screen and health check: failed jobs are solid_queue_failed_executions rows, and work waiting is solid_queue_ready_executions. Retry or discard with SolidQueue::FailedExecution#retry and #discard2. Wire your error tracker into ApplicationJob; the README shows a rescue_from pattern2.

Rollback

Per-class adapters make this cheap. Switch a class back to :sidekiq and new jobs go there. Jobs already in Solid Queue stay in the database, so keep bin/jobs running until they finish. Don't delete the worker classes or uninstall Sidekiq until the drain reads zero.

What stays ugly

  • A small Sidekiq install and Redis for the rate-limited job, and perhaps the batch exports.
  • Two admin screens until the last Sidekiq job is gone.
  • Strict queue order where weights used to be.
  • Retry policies that are shorter than the 25-retry default and now explicit in each class.

If most of the app's value sat in Pro or Enterprise features, I would not migrate it. A well-run Redis costs little, and a half-migrated app costs more than either end state.

Checklist

  • Inventory done: workers, options, middleware, cron, locks, direct Redis calls.
  • Rails 7.1 and Ruby 3.2 or newer.
  • Queue database, pool size and database version checked.
  • Retry and discard rules written per class.
  • Middleware ported to callbacks and tested.
  • Recurring jobs run in one scheduler only.
  • Rate-limited and batch jobs have an explicit decision.
  • Scheduled, retry and dead sets read zero or are deliberately cleared.
  • Mission Control Jobs mounted and protected.
  • Deploy script changed for TERM and shutdown_timeout.

References

  1. Ruby on Rails, "Ruby on Rails 8.0 Release Notes," Ruby on Rails Guides. [Online]. Available: https://guides.rubyonrails.org/8_0_release_notes.html. Accessed: 19 Nov. 2026. ↩

  2. Rails, "rails/solid_queue: Database-backed Active Job backend," GitHub, repository README. [Online]. Available: https://github.com/rails/solid_queue. Accessed: 19 Nov. 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25

  3. Ruby on Rails, "Active Job Basics," Ruby on Rails Guides. [Online]. Available: https://guides.rubyonrails.org/active_job_basics.html. Accessed: 19 Nov. 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8

  4. Ruby on Rails, "ActiveJob::Exceptions::ClassMethods," Ruby on Rails API. [Online]. Available: https://api.rubyonrails.org/classes/ActiveJob/Exceptions/ClassMethods.html. Accessed: 19 Nov. 2026. ↩ ↩2 ↩3

  5. Rails, "Release v1.7.0 - batches," rails/solid_queue, GitHub. [Online]. Available: https://github.com/rails/solid_queue/releases/tag/v1.7.0. Accessed: 19 Nov. 2026. ↩ ↩2

  6. Rails, "rails/mission_control-jobs: Dashboard and Active Job extensions to operate and troubleshoot background jobs," GitHub, repository README. [Online]. Available: https://github.com/rails/mission_control-jobs. Accessed: 19 Nov. 2026. ↩ ↩2

  7. Sidekiq, "Signals," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Signals. Accessed: 19 Nov. 2026. ↩ ↩2

  8. Sidekiq, "Ent Rate Limiting," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Ent-Rate-Limiting. Accessed: 19 Nov. 2026. ↩

  9. Sidekiq, "Batches," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Batches. Accessed: 19 Nov. 2026. ↩

  10. mhenrixon, "sidekiq-unique-jobs: Prevents duplicate Sidekiq jobs," GitHub, repository README. [Online]. Available: https://github.com/mhenrixon/sidekiq-unique-jobs. Accessed: 19 Nov. 2026. ↩

  11. Sidekiq, "Best Practices," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Best-Practices. Accessed: 19 Nov. 2026. ↩ ↩2

  12. Sidekiq, "Active Job," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Active-Job. Accessed: 19 Nov. 2026. ↩

  13. Sidekiq, "Error Handling," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/Error-Handling. Accessed: 19 Nov. 2026. ↩

  14. Sidekiq-Scheduler, "sidekiq-scheduler: Lightweight job scheduler extension for Sidekiq," GitHub, repository README. [Online]. Available: https://github.com/sidekiq-scheduler/sidekiq-scheduler. Accessed: 19 Nov. 2026. ↩

  15. Rails, "rails/activejob/lib/active_job/railtie.rb at 8-1-stable," rails/rails, GitHub. [Online]. Available: https://github.com/rails/rails/blob/8-1-stable/activejob/lib/active_job/railtie.rb. Accessed: 19 Nov. 2026. ↩

  16. Sidekiq, "API," Sidekiq Wiki, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/wiki/API. Accessed: 19 Nov. 2026. ↩ ↩2 ↩3 ↩4

  17. Sidekiq, "sidekiq/lib/sidekiq/api.rb at main," sidekiq/sidekiq, GitHub. [Online]. Available: https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/api.rb. Accessed: 19 Nov. 2026. ↩

Discuss your project

What are you trying to build or decide? A short description of your product, who uses it and where you need help is enough to start.

In our first conversation we can work out whether you need a hands-on engineer, a technical review or a pilot, then agree on scope and timing.