backgroundRB problem (newbie)

Hello,

I am trying to upload and process very large CSV files. For this I am
trying to use backgroundRB. The problem is that I cannot access a
variable inside my worker until the methods have finished even though
I am calling the method asynchronously.

My controller:

def authenticate_import
         @dataset = Dataset.new params[:dataset]
         @dataset.campaign = @campaign
        #Check the normal info is valid
        if !@dataset.valid?
            render :update do |page|
                page.replace_html 'import', :action => 'import'
            end
        else
      @key = Time.now.to_i
            @worker = MiddleMan.worker(:dataset_worker, @key)
      @worker.async_process_csv(:arg => @dataset)
       render :update do |page|
                page.replace_html 'progress', :partial =>
'progress_meter', :object => @key, :locals => {:progress =>
@worker.progress}
            end
        end
    end

This should return the variable progress to the partial which at the
moment simply displays the local progress.

My worker:

class DatasetWorker < BackgrounDRb::MetaWorker
  set_worker_name :dataset_worker

  attr_accessor :progress

  def create(args = nil)
    logger.info "Dataset Worker setup"
  @progress = 0
  end

  def process_csv(dataset)
    logger.info "Process CSV method called"
    logger.info "Dataset is a #{dataset.type}"

    logger.info "Dataset: begin parsing file"
      dataset.process_file
      @progress = 30
    logger.info "Dataset: parsing complete"
    logger.info "Dataset: Creating table in db"
      if dataset.save
        table = dataset.table
        total = table.size
            begin
                #Create the fields first, otherwise we would have to
check whether they had been created
                  #for every row were inserting (alot of SQL calls for
large CSV files)
                  row_cnt = 0
                  table.headers.each do |field|
                   DatasetField.create({:dataset =>
dataset, :dataset_field_name => field})
                end
                #reload fields
                dataset.dataset_fields(true)
                table.by_row!.each do |row|
                      dataset.create_row row
                    row_cnt++
                      @progress = ((row_cnt/total)*70)+30
                end
           end
    end
  logger.info "Dataset: Finished creating table in db"
  end

end

I am trying to access the variable progress so that I can display it.
Then I can periodically update the progress by using the job key. The
problem is the Ajax requests doesn't finish until the method
process_csv has finished.

Any help would be greatly appreciated

Hello,

I am trying to upload and process very large CSV files. For this I am
trying to use backgroundRB. The problem is that I cannot access a
variable inside my worker until the methods have finished even though
I am calling the method asynchronously.

Problem is by default worker is synchronous and uses event driven
network programming, so next event is not picked until existing
handler returns. But, if you #defer process_csv inside a thread, next
call to @worker.progress should return as you expect (for using #defer
method inside worker, refer backgroundrb.rubyforge.org).

Also, if you are going to use threads, then you need to protect
@progress accessor with a mutex or something, so as concurrent
threads, update it synchronously.

Another alternative is to use, bakcgroundrb with memcache for caching
result objects (such as progress bar and stuff). From worker just do:

cache[:progress_percent] = 80

and from rails:

MiddleMan.worker(:foo_worker).ask_result(:progress_percent)

(refer official docs for more info).