How to configure solid_queue with two queues, one for general purpose "one at a time" and another for a rate limited API?

rubyonrails-talk
1

You can provide a single queue, or a list of queues as an array. Jobs will be polled from those queues in order, so for example, with [ real_time, background ], no jobs will be taken from background unless there aren’t any more jobs waiting in real_time.

workers:
  - queues: "*"
    threads: 5
    processes: 1
    polling_interval: 0.1

What I’m looking for is to have two queues:

  1. General purpose, 1 at a time concurrency.
  2. Rate limited, OpenAI API, 1 at a time concurrency.

I want an OpenAI queue job to run one at a time but never have to wait for the general purpose queue to empty.

For example in other stacks like Elixir w/Oban, I can define the concurrency of each queue but that does not affect the priority of the queue. All three queues run at the same time, just at different concurrency limits.

queues: [default: 10, howlongtobeat: 1, posthog: 1]

Is something like this possible with solid_queue?

2

You can limit concurrency using the limits_concurrency method without even needing a separate queue (SolidQueue docs):

# preferred solution
class RateLimitedJob < ApplicationJob
  limits_concurrency to: 1, key: :openai_api
end

This would likely be the simplest approach. If there’s a particular reason why you would prefer having separate queues and not using the limits_concurrency method, you could do something like this:

# alternative solution
class RateLimitedJob < ApplicationJob
  queue_as: :openai_api
end

# queue.yml
...
workers: 
  - queues: <%= ENV.fetch("QUEUES", "*") %>
    threads: <%= ENV.fetch("THREADS", 3) %>
    processes: <%= ENV.fetch("JOB_CONCURRENCY", 1) %>
    polling_interval: 0.1

and run your separate workers with:

# throttled job runner
QUEUES=openai_api THREADS=1 JOB_CONCURRENCY=1 bin/jobs
# normal job runner
QUEUES=<your other queues excluding openai_api> bin/jobs
3

If you do want separate queues, the README has an example:

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300
  workers:
    - queues: "*"
      threads: 3
      polling_interval: 2
    - queues: [ real_time, background ]
      threads: 5
      polling_interval: 0.1
      processes: 3