Thanks glennswest, im relatively new to rails. Whilst i think i
understood what you said can you (or anyone else) elaborate furhter on
the points below? I really appreciated your help.
Just thinking, your scrape should probably be in a worker,
when you say a worker i take it you mean some temporary database?
Depending on what your using, you configure it to be a temp
Then in your search window you can do ajax based updated from the
From the above do you mean whilst im scraping results from sites, when
one sites results get added to the db and i go off scraping another
sites results, i can simultaneously show the results that were just
added to the screen?
With the ability to then clear up the cache.
after i get all the results and display them to the screen i can then
clear the table?
You get more concurrency,
Wasnt too sure what you meant by this but thats because im fresh to
rails and cant gather from the context.
ahh so if whilst im scraping and simultaneously presenting already
scraped data from the db, the user decides to cancel the request, via
Think this would scale and be more responsive
In general how fast/slow is it to update a table with around 1000
results? is it fast enough to handle this situation? Id prefer to stick
the objects in a temporary db because then id get to use existing
activerecord methods and mysql statements. Im just worrying about the