[RESQUE]Question about queues

rubyonrails-talk
1

I run Resque with x 'general' queues. With 'general' I mean a queue that may run every type of Worker. Or in the Rake command: COUNT=x QUEUE=* rake resque:work

When a worker runs a job of a specific class, it may not run another job of the same class simultaniously. This job has to wait until the previous job has finished before it may start. BTW The reason jobs may not run simultaniously is that the jobs do extrernal HTTP-requests. Too many external HTTP-requests for one class may occur a too heavy load for the service this class is using.

The most easy solutaion is to do just this for every class I need a worker: QUEUE=<class> rake resque:work The problem with this solution is that it consumes a lot of memory. At the moment I have 30 different classes and it is still growing.

Another solution is to let several classes share one queue like this: QUEUE=<class1>,<class2>,<class3> rake resque:work The problem with this is when a job of e.g. class1 hangs, jobs of class2 and class3 won't run either. Exiting the job when it runs too long isn't an option either, because I never know how long a normal job will run.

So that's why I'm coming to the solution I described at the start of this post. Create an x number of workers that may run every type of job, BUT when a job of specific type is running, the workers may not run a second job of that type.

My problem is that I don't know how to implement this in Resque. Any ideas?

2

I just searched through the Resque plugins available at https://github.com/defunkt/resque/wiki/plugins but I can't find a plugin that fit my needs.

3

Have you looked at resque-lock or resque-loner?

4

My queues consists of jobs with different arguments.

resque-lock is mentioned to add just one job per queue. That isn't going to work, because I have different jobs for one queue.

resque-loner is mentioned to have one unique job on the queue. That is not my problem, My problem is that a job of a unique class always have to use the same worker, even when there are multiple workers available. If there are multiple workers available, the job has to be queued to be runned by one specific worker, even if the other workers are in idle.