I am trying to find a way to use the new twitter streaming API
(continuous XML or JSON stream) in a ruby/rails project.
I want to grab the stream on my server, process it with different
algorithms, and then output the result to whoever has the flash front
end and is asking for it.
I was suggested the xmpp protocol, but I might not have time to
investigate this path... So I figured I could simply put a timer on
the client and ask periodically the server for a bunch of "processed
stream" through a regular http request...
I therefore believe that I will always have a process running (at
least as soon as someone requests something at least). My algorithms
are not a problem, I am getting the stream in real-time from a ruby
script, BUT the part that leaves me clueless is how can I do all this
processing only in a fixed amount of memory on the server and send it
whenever someone asks for it. Of course, at one point the old
processed elements get overwritten by new ones.
How can I pass information from the memory being handled by a daemon
to a rails app?
Perhaps I simply don't understand something and I am asking the wrong
question, but any pointers would be greatly appreciated.