Queue Abstraction Layer

I want to take a moment to talk about the notion of a queue abstraction layer in Rails and put a stake in the ground with my opinions on the topic while at the same time getting some feedback.

My understanding is this: at the moment ActiveWorker (which can be seen in this repo: http://github.com/joshbuddy/rails ) is the current track for how to integrate workers (and queues to an extent). There's nothing wrong with this approach, however my position is that workers != queues, and I don't necessarily want to define my workers in the context of Rails, yet I do want an abstraction layer for queues. The current design of ActiveWorker has the queues essentially as adapters, similar to how ActiveRecord works, so I can understand leaving this as part of ActiveWorker, but I do wonder if it's a good idea to couple these things together. Personally I don't think it is but then again others may disagree.

Here's my proposal: there should be a queue abstraction layer (call it ActiveQueue if you like) with the following high-level requirements:

* It should provide a generic interface to queues (messages queues and publish/subscribe) * It should provide a standard interface for drivers to implement and should provide a few basic driver implementations * It should allow other drivers to be added in through simple configuration (config/queue.yml) * It should define an interface for serializing messages (and provide JSON and XML serialization as the default?) * It should NOT define how to handle work units (i.e. this is not for defining asynchronous workers) * It should be used under the hood by ActiveWorker

I actually have a need for this right now and would be willing to implement it to scratch my own itch, but I didn't want to do so without first getting some feedback from the core community to see if this is both interesting and useful.

Thoughts?

Sincerely, Anthony Eden

+1 on the concept ... I feel that Rails has been without this for a while. With the recent surge of activity on queue services, I think this is an awesome suggestion ... it would be great to have a standard Rails-ish way of setting up multiple work queues.

I think that a good starting point for where to look for inspiration is http://github.com/defunkt/resque#readme ... which has a ton of really good design choices.

If there was a common interface to queues such as Resque that could be abstracted in front of any choice of queue backend, I feel like it would be a huge win for Rails.

Paul

+1

I would love to see the related scheduled job be part of this as well since some processes can only have a single instance running at a time.

Interesting ideas.

I think it be "ActionQueue" since its not a model layer (not sure).

Maybe both belong in the same "pack" similar to ActionPack.

So we have some root package called "ActionBackgroundShit" and it contains AW and AQ.

actionbackgroundshit/lib/action_worker/* actionbackgroundshit/lib/action_queue/*

Queues != Model. Josh, you are right indeed. +1 from me for AW & AQ :slight_smile:

Cheers,

Amol

Seems to me it’s analogous to caching from a service-oriented architecture pov - ActiveSupport::Queue like ActiveSupport::Cache ? Interface/implements approach? AMPQ-compliant interface? Starling? Also suggest that there may be different levels of the interface, starting with very simple transient push/pop all the way to full AMPQ.

This would allow work around ActiveWorker to focus exclusively on worker challenges like polling, starvation, backoff, sleep, wake, deployment, running in context, etc.

m