Hey there,
I've just tried to load a massive CSV fixture file of about 2700000 lines (cites, locations etc). Not surprisingly it died half way through.
Does anyone know how to load such an amount of data? Maybe splitting it into multiple files?
Sorry for such a newb question... but, well, I am (to rails).
John
El 8 Apr 2011, a les 22:45, John Wright <lists@ruby-forum.com> va escriure:
Hey there,
I've just tried to load a massive CSV fixture file of about 2700000 lines (cites, locations etc). Not surprisingly it died half way through.
Does anyone know how to load such an amount of data? Maybe splitting it into multiple files?
Sorry for such a newb question... but, well, I am (to rails).
Do you need those many records for your test suite?
John Wright wrote in post #991838:
I've just tried to load a massive CSV fixture file of about 2700000 lines (cites, locations etc). Not surprisingly it died half way through.
Does anyone know how to load such an amount of data? Maybe splitting it into multiple files?
Sorry for such a newb question... but, well, I am (to rails).
Don use fixtures, not even when just starting out. Instead choose one of the excellent test data factory gems to generate test data. And, only generate test data specifically for the tests you write. You're not going to write 2700000 test on city data. Just a few examples are all you'll need for your test suite. Generate those as you need them with your factory.
Does anyone know how to load such an amount of data? Maybe splitting it into multiple files?
If you are loading it into a MySQL database, use the LOAD DATA INFILE command. It's very fast and pretty flexible in terms of allowing you to define the record separators, file separators, and field mappings.
- D