Had a very similar task, a progress bar updating in the browser while some long-running task is running on the server. WEBrick can only server one request at a time, which explains the behavior you describe . . . all the “update progress bar” requests get processed after the long-running action is completed. You may be interested in Apache and mod_fcgid . . . you can run multiple Rails dispatchers (basically a separate ruby instance for each, though they are not really ruby processes, they are threads which are managed by mod_fcgid)
Then your progress bar can work, but there’s still a caveat: these threads dont share any memory because that was not part of the design of Rails (this is what is mean by “not thread-safe”). The way to do it is to put the progress bar value in the session, and in your long-running action, call session.update after setting the progress value. The progress meter is then just some AJAX polling (PeriodicallyCallRemote) calling an action which renders the progress meter. If you dont call session.update, Rails calls it for you after the action completes, but in this case you will want to call it, since you need that progress value to be available to ALL the mod_fcgid threads before your long-running action completes.
Hope that helps.