I've finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this..
What I have tried right now is to call a javascript function on submit
of the form specifying the file to import. Every 5 seconds this
function creates an Ajax request to a "update_progress" method
inside my controller. What I'm seeing is that all the Ajax requests
are not called until AFTER my method "csv_import" that parses the csv
file completes (rails seems to be too busy parsing).. so all I get is
0% and then jumps to 100%. I've also tried adding a render statement
to the "update_progress" method inside the loop that parses the csv
file but this is a problem since rails only lets you render once in a
method (I'm already rendering the results at the end). Does any one
know a better way for me to display to the user the progress of my
"csv_import" method.
Basic outline of method "csv_import" and "update_progress":
$total_lines_read = 0
def csv_import
open file
read & add record for each line to the db
increment $total_lines_read
render the results
end
def update_progress
render javascript function to update the bar sending it
$total_lines_read
end
I've finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this..
Oh yeah. HTTP requires very narrow events. To fix this, you must stretch one logical event out over many physical events.
Try this:
- the "Go" button starts a periodically_call_remote
- that calls an action
- the action opens the CSV,
- reads one batch from it (say 20 lines)
- stash the current record count in the session
- returns nothing
- the periodically_call_remote ticks the progress bar
- the periodically_call_remote stops when the action
- returns JavaScript assigning 'false' to a global variable
- the periodically_call_remote's condition is that variable
That sounds like a lot of work, but the alternatives are all worse!
And if you progress, you should have a cancel button. My system makes your reader function easier to interrupt.
Another option is to use backgroundrb to process the csv file outside
the current rails process. This has the advantage of not blocking web
requests to that process because your app is processing the file, as
well as allowing you to check in on the job periodically using the
worker id to see how it's going and to update your progress bar.
then again, getting backgroundrb running for this particular problem
may be overkill
When you control the loop, a simple solution is "action chunking" :
cut your big task/action into smaller chunks, that chain/call themself
automatically, and display a little visual feedback between each
transition.
then again, getting backgroundrb running for this particular problem
may be overkill
hth, Matt
That's why I didn't mention backgroundrb. (We abuse that all the time, at my day-job, to avoid action-chunking!)
If the OP learned they could install a fragile deployment-nightmare as an excuse not to split their inner loop up into action chunks, they might behave the same way Java programmers do when they hear that threading in Java is "easy"...
I've finished creating a csv import ability in my web application that
parses a csv file and adds records into the MySQL db. I figuired it
would be nice to have a progress bar for this import process.
Unfortunately I have run into issues trying to implement this..
I had to do almost exactly this, I screen casted it and documented it here:
