I want to take a moment to talk about the notion of a queue abstraction layer in Rails and put a stake in the ground with my opinions on the topic while at the same time getting some feedback.
My understanding is this: at the moment ActiveWorker (which can be seen in this repo: http://github.com/joshbuddy/rails ) is the current track for how to integrate workers (and queues to an extent). There's nothing wrong with this approach, however my position is that workers != queues, and I don't necessarily want to define my workers in the context of Rails, yet I do want an abstraction layer for queues. The current design of ActiveWorker has the queues essentially as adapters, similar to how ActiveRecord works, so I can understand leaving this as part of ActiveWorker, but I do wonder if it's a good idea to couple these things together. Personally I don't think it is but then again others may disagree.
Here's my proposal: there should be a queue abstraction layer (call it ActiveQueue if you like) with the following high-level requirements:
* It should provide a generic interface to queues (messages queues and publish/subscribe) * It should provide a standard interface for drivers to implement and should provide a few basic driver implementations * It should allow other drivers to be added in through simple configuration (config/queue.yml) * It should define an interface for serializing messages (and provide JSON and XML serialization as the default?) * It should NOT define how to handle work units (i.e. this is not for defining asynchronous workers) * It should be used under the hood by ActiveWorker
I actually have a need for this right now and would be willing to implement it to scratch my own itch, but I didn't want to do so without first getting some feedback from the core community to see if this is both interesting and useful.
Thoughts?
Sincerely, Anthony Eden