Introducing Solid Queue for Background Jobs

Things are changing on Ruby on Rails 8.0! Solid Queue and Solid Cache will be the new default gems for background jobs and caching for Rails 8. In today’s article, we will focus on Solid Queue, what it is, how it works, and basic setup instructions. By the end of this post, you should have a better understanding of the basic technologies that will come with Rails 8 projects and be a bit more well-equipped to decide whether to use the Database or Redis to power your background jobs for your project.

What’s Solid Queue?

“Solid Queue is a DB-based queuing backend for Active Job, designed with simplicity and performance in mind.” ref

Due to its simplicity, Solid Queue can be used with the most commonly used SQL databases such as PostgreSQL and MySQL. But simplicity doesn’t mean it’s weak. Solid Queue supports delayed jobs, queue priority, bulk enqueuing, and more!

How it works?

Unlike Sidekiq and Resque, which use Redis for managing background job queues, Solid Queue centralizes this process within the project’s database. During configuration, it generates multiple tables for storing ready, enqueued, and failed jobs, and more. Although this creates additional "noise" in the database schema, it provides transparency for background jobs, enabling project managers to directly monitor and modify jobs without the need for any external tools.

One of the most impressive characteristics of Solid Queue is its native multiprocessing capability. With minimal setup required, it provides scalability and improved performance for small to medium projects without increasing infrastructure costs and maintenance.

Normally, workers wouldn’t be able to “compete” for the next job to be processed, since the table would be locked. But what allows Solid Queue to work is the usage of FOR UPDATE SKIP LOCKED, which allows multiple processes to access the table concurrently.

Solid Queue setup:

Add Solid Queue gem to your gemfile:

bundle add solid_queue

Then run Solid Queue’s installer to create the necessary migrations and add the default configuration file:

bundle exec rails generate solid_queue:install

Update your queue adapter adding the following line to your development.rb.

config.active_job.queue_adapter = :solid_queue

Finally run migrations:

bundle exec rails db:migrate

And we’re all set! To start your workers run:

bundle exec rake solid_queue:start

How to configure Solid Queue?

After setup, the following configurations file is created at config/solid_queue.yml:

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300
  workers:
    - queues: "*"
      threads: 3
      polling_interval: 2
    - queues: [ real_time, background ]
      threads: 5
      polling_interval: 0.1
      processes: 3

We won’t talk about every single configuration option, since this can be found on Solid Queue’s documentation, but let’s point out some interesting elements of its configuration.

Dispatchers are responsible for selecting which scheduled jobs will be ready to be picked up by workers, moving them to the correct database tables, as well as concurrency management.
Workers look for jobs on the solid_queue_ready_executions table and process them.
Queues define specific types of jobs to be handled, allowing you to prioritize and allocate resources efficiently based on the nature and importance of different tasks within your application.
The processes option determines the number of worker processes that Solid Queue can spawn to execute jobs concurrently. Each process operates independently and can handle multiple jobs simultaneously.
Threads specifies the maximum number of concurrent threads within each worker process that Solid Queue can utilize for job execution.

Solid Queue vs Redis-Based background jobs:

When comparing Solid Queue with Redis-based background job processing in Rails, several factors come into play. Let’s examine the arguments for each:

Solid Queue:

Simplicity: The setup process is simple and it directly integrates within a Rails application. The level of setup effort is minimal and enables fast integration of a Rails application with a background job processing system.
No external dependencies: Solid Queue does not require any other external services, such as Redis, for it to work. This makes deploying and running it easy and is convenient for smaller applications or projects that don’t want to have an additional dependency.
Flexibility: Solid Queue allows developers to fine-tune various aspects of job processing, including concurrency control, error handling, and middleware integration. This flexibility enables customization to meet specific application requirements without being tied to the constraints of an external system like Redis.

Redis-based Background Jobs:

Scalability: Redis has a reputation for being scalable and integrating high-performance data structures, which is why it is a natural option for distributed systems and large-scale applications. Redis-backed job processing can use its unique features such as pub/sub, lists, and sets to create resilient job queues capable of handling massive workloads
Advanced Features: Redis offers a wide range of advanced features beyond basic job queuing, including data caching, pub/sub messaging, and real-time analytics. Leveraging these capabilities can enable developers to build more sophisticated and feature-rich applications beyond just background job processing.
Ecosystem: Redis is a popular system with a strong community base and extensive third-party documentation and libraries. This ecosystem provides access to additional tools, integrations, and best practices for building scalable and robust apps

In conclusion, the choice between Solid Queue and Redis-based solutions depends on factors such as application complexity, scalability needs, and developer preferences. By understanding the nuances of each approach, Rails developers can make informed decisions to optimize background job processing in their projects.

Wrapping up

And that’s it! In this article, we delved into the evolving landscape of Rails 8.0, highlighting the transition to Solid Queue as the new default gem for background jobs.
Our goal was to provide a friendly introduction to the gem and discuss core functionalities and characteristics to help you make better-informed technical decisions on your projects.