backgroundRB problem (newbie)

rubyonrails-talk
1

Hello,

I am trying to upload and process very large CSV files. For this I am trying to use backgroundRB. The problem is that I cannot access a variable inside my worker until the methods have finished even though I am calling the method asynchronously.

My controller:

def authenticate_import          @dataset = Dataset.new params[:dataset]          @dataset.campaign = @campaign         #Check the normal info is valid         if !@dataset.valid?             render :update do |page|                 page.replace_html 'import', :action => 'import'             end         else       @key = Time.now.to_i             @worker = MiddleMan.worker(:dataset_worker, @key)       @worker.async_process_csv(:arg => @dataset)        render :update do |page|                 page.replace_html 'progress', :partial => 'progress_meter', :object => @key, :locals => {:progress => @worker.progress}             end         end     end

This should return the variable progress to the partial which at the moment simply displays the local progress.

My worker:

class DatasetWorker < BackgrounDRb::MetaWorker   set_worker_name :dataset_worker

  attr_accessor :progress

  def create(args = nil)     logger.info "Dataset Worker setup"   @progress = 0   end

  def process_csv(dataset)     logger.info "Process CSV method called"     logger.info "Dataset is a #{dataset.type}"

    logger.info "Dataset: begin parsing file"       dataset.process_file       @progress = 30     logger.info "Dataset: parsing complete"     logger.info "Dataset: Creating table in db"       if dataset.save         table = dataset.table         total = table.size             begin                 #Create the fields first, otherwise we would have to check whether they had been created                   #for every row were inserting (alot of SQL calls for large CSV files)                   row_cnt = 0                   table.headers.each do |field|                    DatasetField.create({:dataset => dataset, :dataset_field_name => field})                 end                 #reload fields                 dataset.dataset_fields(true)                 table.by_row!.each do |row|                       dataset.create_row row                     row_cnt++                       @progress = ((row_cnt/total)*70)+30                 end            end     end   logger.info "Dataset: Finished creating table in db"   end

end

I am trying to access the variable progress so that I can display it. Then I can periodically update the progress by using the job key. The problem is the Ajax requests doesn't finish until the method process_csv has finished.

Any help would be greatly appreciated

2

Hello,

I am trying to upload and process very large CSV files. For this I am trying to use backgroundRB. The problem is that I cannot access a variable inside my worker until the methods have finished even though I am calling the method asynchronously.

Problem is by default worker is synchronous and uses event driven network programming, so next event is not picked until existing handler returns. But, if you #defer process_csv inside a thread, next call to @worker.progress should return as you expect (for using #defer method inside worker, refer backgroundrb.rubyforge.org).

Also, if you are going to use threads, then you need to protect @progress accessor with a mutex or something, so as concurrent threads, update it synchronously.

Another alternative is to use, bakcgroundrb with memcache for caching result objects (such as progress bar and stuff). From worker just do:

cache[:progress_percent] = 80

and from rails:

MiddleMan.worker(:foo_worker).ask_result(:progress_percent)

(refer official docs for more info).