Non-blocking "send_data `foo`"

Hello list,

The subject pretty much says it all: I'm looking for a way to send the
output from a long-running, slowly-outputting system command to the
browser without the app hanging until that system command completes.
I've tried the obvious idioms (see below), but there doesn't seem to be
a way to prevent the application blocking on the system command. When
(eventually) foo completes, the data is sent to the browser, but the
poor user has to sit watching the egg-timer until all of the data has
been slurped into memory by rails before it starts the send. I'd like
the data to start streaming to the browser as soon as it becomes
available from the system command.

I have tried:
   backticks
   IO.popen
   Proc.new { ... File.popen ... }

Perhaps I need something other than send_data? Is there a way to jump
out of rails to use exec from the controller ("application failed to
start")?

I am considering POSTing to a small bespoke CGI. Or maybe I could do
something cunning with Net::SSH channels? The first of these will
certainly work, but neither seem very satisfactory. What am I missing?

Thanks for any enlightment,

Martin.

Martin Carpenter wrote:

Hello list,

The subject pretty much says it all: I'm looking for a way to send the
output from a long-running, slowly-outputting system command to the
browser without the app hanging until that system command completes.
I've tried the obvious idioms (see below), but there doesn't seem to be
a way to prevent the application blocking on the system command. When
(eventually) foo completes, the data is sent to the browser, but the
poor user has to sit watching the egg-timer until all of the data has
been slurped into memory by rails before it starts the send. I'd like
the data to start streaming to the browser as soon as it becomes
available from the system command.

I have tried:
   backticks
   IO.popen
   Proc.new { ... File.popen ... }

Perhaps I need something other than send_data? Is there a way to jump
out of rails to use exec from the controller ("application failed to
start")?

I am considering POSTing to a small bespoke CGI. Or maybe I could do
something cunning with Net::SSH channels? The first of these will
certainly work, but neither seem very satisfactory. What am I missing?

Thanks for any enlightment,

Martin.

Pretty sure that rails doesn't do streaming of any sort.

You could spin up a BackgroundDRB process, and send the user to a page
with a big happy animated gif that says loading. And refresh, or make a
little AJAX call, every few seconds. If the job is done, get its data,
show it to the user.

Or do some sort of polling to get the computed values from your
background process as it chugs along.

Someone may know of a way, but I don't think rails is capable of
streaming much of anything.

Hello list,

The subject pretty much says it all: I'm looking for a way to send the
output from a long-running, slowly-outputting system command to the
browser without the app hanging until that system command completes.
I've tried the obvious idioms (see below), but there doesn't seem to
be
a way to prevent the application blocking on the system command. When
(eventually) foo completes, the data is sent to the browser, but the
poor user has to sit watching the egg-timer until all of the data has
been slurped into memory by rails before it starts the send. I'd like
the data to start streaming to the browser as soon as it becomes
available from the system command.

I have tried:
  backticks
  IO.popen
  Proc.new { ... File.popen ... }

Perhaps I need something other than send_data? Is there a way to jump
out of rails to use exec from the controller ("application failed to
start")?

Use a mongrel handler? That won't block your app (but ties you to
mongrel)

Fred

Thanks Fred, good suggestion (although I also don't particularly like
being tied to mongrel, it does happen to be what we're using, and it
gets me over the hump for now). The other big help was Alex's
re-emphasis of the word "streaming" which got me thinking about audio
and video streaming, and turned up this nice blog entry:

http://blog.rubyenrails.nl/articles/2008/02/23/breaking-out-of-rails-mvc-with-mongrel-handlers

Problem solved!

I have a couple of smaller issues if there's anyone with any deeper
mongrel handler knowledge here - for instance, I would like to
redirect_to a controller (possibly setting flash[:error] on the way) if
there's a problem, but redirect_to is undefined. I'd also like to DRY-up
my handler since I've reimplemented (copy/paste) a couple of small
private methods from my controller - they could be moved to the model,
but then I can't see how to call those from the mongrel handler either?
The mongrel documentation is a little... sparse :-/

Thanks again,

Martin.

The mongrel handler is just ruby code in the same memory space as the
rest of your app. Put the shared code in a module and include it in
both handler & controller
Your mongrel handler isn't a controller so all the niceties like the
flash and redirect_to are gone. If you want to redirect then you'll
need to generate the 302 response yourself. Be wary about calling
code from your app, there are bits of rails that are not thread safe.

Fred

All fixed - thanks for the assistance!

Martin.