Splitting up a long running task with delayed job

I have a method on my model that looks like this.

def self.itunes_update   # don't run it if it's already running   if Setting.first.update_lock?     puts "updates locked"   # it's not running right now so run it   else     all_apps = self.all     all_apps.each do |app|       # do stuff with each app     end   # job done, clear the lock   end end

It works fine, but it processes 10,000 records and takes about an hour to run on heroku. I originally just had a rake task calling it every 3 hours but heroku were SIGTERMing it and suggested that I move it to delayed_job, which I have done like this..


it's working fine.

They do however suggest that I process this in batches or smaller chunks. How would I do that, and does that sound like a good idea?

Also - do I need to do the locking any more, I think delayed job handles it for me - that said, guess it doesn't do any harm?



anyone done anything like this before?

bingo bob wrote in post #989039:

anyone done anything like this before?

What if you select 100 records per pass in an oldest-update == first-selected order?

model.Find(:all, :order => "updated_at DESC", :limit => 100)

I'm not sure this is what you mean, but I use a daemon for anything that would make the browser respond slowly. I have a table "delayed_tasks" and the controller just writes a row to that table and finishes. Then the daemon looks for rows in the table and handles them behind the scenes and deletes them when handled.

For instance, I do that for sending email because it takes a number of seconds for email to get delivered.

If it is something the user is waiting for, the browser can periodically do an ajax call to see if the results are ready.