Queue Abstraction Layer

rubyonrails-core
#1

I want to take a moment to talk about the notion of a queue
abstraction layer in Rails and put a stake in the ground with my
opinions on the topic while at the same time getting some feedback.

My understanding is this: at the moment ActiveWorker (which can be
seen in this repo: http://github.com/joshbuddy/rails ) is the current
track for how to integrate workers (and queues to an extent). There's
nothing wrong with this approach, however my position is that workers
!= queues, and I don't necessarily want to define my workers in the
context of Rails, yet I do want an abstraction layer for queues. The
current design of ActiveWorker has the queues essentially as adapters,
similar to how ActiveRecord works, so I can understand leaving this as
part of ActiveWorker, but I do wonder if it's a good idea to couple
these things together. Personally I don't think it is but then again
others may disagree.

Here's my proposal: there should be a queue abstraction layer (call it
ActiveQueue if you like) with the following high-level requirements:

* It should provide a generic interface to queues (messages queues and
publish/subscribe)
* It should provide a standard interface for drivers to implement and
should provide a few basic driver implementations
* It should allow other drivers to be added in through simple
configuration (config/queue.yml)
* It should define an interface for serializing messages (and provide
JSON and XML serialization as the default?)
* It should NOT define how to handle work units (i.e. this is not for
defining asynchronous workers)
* It should be used under the hood by ActiveWorker

I actually have a need for this right now and would be willing to
implement it to scratch my own itch, but I didn't want to do so
without first getting some feedback from the core community to see if
this is both interesting and useful.

Thoughts?

Sincerely,
Anthony Eden

#2

+1 on the concept ... I feel that Rails has been without this for a
while. With the recent surge of activity on queue services, I think
this is an awesome suggestion ... it would be great to have a standard
Rails-ish way of setting up multiple work queues.

I think that a good starting point for where to look for inspiration
is http://github.com/defunkt/resque#readme ... which has a ton of
really good design choices.

If there was a common interface to queues such as Resque that could be
abstracted in front of any choice of queue backend, I feel like it
would be a huge win for Rails.

Paul

#3

+1

I would love to see the related scheduled job be part of this as well
since some processes can only have a single instance running at a
time.

#4

Interesting ideas.

I think it be "ActionQueue" since its not a model layer (not sure).

Maybe both belong in the same "pack" similar to ActionPack.

So we have some root package called "ActionBackgroundShit" and it
contains AW and AQ.

actionbackgroundshit/lib/action_worker/*
actionbackgroundshit/lib/action_queue/*

#5

Queues != Model. Josh, you are right indeed. +1 from me for AW & AQ :slight_smile:

Cheers,

Amol

#6

Seems to me it’s analogous to caching from a service-oriented architecture pov - ActiveSupport::Queue like ActiveSupport::Cache ? Interface/implements approach? AMPQ-compliant interface? Starling? Also suggest that there may be different levels of the interface, starting with very simple transient push/pop all the way to full AMPQ.

This would allow work around ActiveWorker to focus exclusively on worker challenges like polling, starvation, backoff, sleep, wake, deployment, running in context, etc.

m