Hi guys, I wanted to discuss the new Queueing API for those of us who
are implementing an out-of-process version. In my case, I write
Sidekiq  and would like to support the new API once Rails 4 is
released. My issue is that because the API is object-oriented rather
than message-oriented, implementation of out-of-process workers is
The API is Queue#push(job) where job has a run method. Ruby doesn't
have a great solution for serializing a Ruby object across the wire.
Marshal limits the API to Ruby solutions (which rules out RabbitMQ, et
al), JSON can't fully serialize Ruby objects (e.g. symbols) and YAML
has a number of issues in practice that make it painful to use (e.g.
see the monkeypatches DelayedJob has to use ).
I don't understand what you mean by this. Marshal returns a string. If
your producers and consumers are both Ruby, why would this rule out
anything? Are you saying you want to mix languages between producers
So I love the simplicity of the API but think it will lead to painful
implementation issues. What do you think about defining a simpler
message format that can be fully serialized and deserialized via JSON
/ YAML / etc instead of using a Ruby object?
I don't think the serialization format is something that Rails should
define. It's an implementation detail of the queuing system, and is
something a user must understand when using a queue.
* an in-memory queue has the advantage doesn't require serialization
but is volatile. But maybe that's all the programmer needs.
* a DRb based queue can be distributed and uses marshal, so the user
doesn't need to understand serialization as much.
* An object that has references to an IO object must take precautions
when being serialized.
The other problem is that maybe the JSON format that sidekiq requires
could possibly be different than the JSON format that some other queuing
tl;dr every queue has unique aspects regarding transactions, wire
protocol, and storage facility. Users should take this in to account
when selecting a queue for their application's requirements.