Hi everyone,
I have an application where an essential part is communicating with a remote API via SOAP. I want to be able to run 3-4 of these tasks at once, and one task might take anything from one minute to several hours, depending on the size of the data being sent. The app relies heavily on ActiveRecord fetching fetching data (in_batches) and updating it once the response from the API has been received.
I'm using Ruby 1.8.6 and Rails 2.3.4.
In an earlier version of this application (written in PHP), this was set up as a cron job which called the relevant scripts via curl. This was obviously not the best implementation, and so it was decided to move to Rails.
After having done quite a bit of research, I can't for the life of me figure out what the best approach to handle these long-running tasks would be. Initially I was going to set them up as Rake tasks, and call them with cron every minute. I had a database table with active tasks and would only start the task if less than 4 tasks were already running.
Then I discovered Daemons, and was trying to figure out if that would be a better approach to my issue. And then, I discovered Delayed::Jobs, which also could be helpful.
Naturally, I want to perform these tasks as efficiently as possible, with the least amount of resources. The tasks will be more-or-less around the clock, so I would probably rather have it run continually (like a Daemon) than having to reload the libraries every time (which I would have to do with Rake). On the other hand, I'm not sure how a Daemon would handle concurrent tasks--threading, possibly?
With all of the different options out there, I'm getting more and more confused by the minute. Help, anyone?
Best regards, Sebastian
