Advice RE: Very long solid queue jobs (hours - days)

rubyonrails-talk
1

We are considering migrating our custom queue processor (of about 15 years) to SolidQueue. The biggest catch is that many of our jobs take minutes, some take hours, and a few take days. (these are large data imports and/or processing of molecular data)

Our existing system is configured to make sure these jobs run to completion - even through upgrades to the core app or when their job manager dies/goes away; only giving up when manually KILL’d (though there is some process monitoring to KILL anything that looks like it’s spinning/growing in memory unbounded).

Is this reasonable/easy to do with SolidQueue? Has anyone else done this/have any advice?

1 Like
2

Maybe versioning and feature flags can helpful for your case

# pseudo code to expain
class ExampleController
  def create
	if ENV['USE_NEW_WORKER']
      NewSolidQueueWorker.perform_later(params)
    else
      OldCustomJobWorker.perform_later(params)
    end
  end
end

Also maybe helpful:

3
4

SolidQueue can handle long jobs, but for multi-hour/day tasks, consider external process management (e.g., systemd or Kubernetes) for resilience.

5

If you have such a problem you want to have your jobs resumable or restartable. This is not something any ActiveJob adapters handle for you.

You may want to look into job-iteration, active-job-continuation, acidic_job or stepper_motor to split these jobs into smaller steps or to make them iterate. I would say from what you are telling either ..-continuation or job-iteration will fit the use case.