Importing Large Data Sets into Rails App

I am building an app for a client that will require importing large amounts (~2GB) of zipcode-specific data from Excel files. I was just wondering what you folks thought the best solution for importing the data to the database would be? Currently I'm planning on converting the Excel data to .csv and writing a script that will load from an arbitrary number of .csv diles in a directory, but that will take hours to complete. Is there a better way?


Michael Bleigh

Well, you will not want to create an AR object for each row, because,
like you said, that would take hours and hours. You should use the
actual DB connection that AR gives you and do multiple inserts with
straight SQL. If you are using MySQL the best way to load that amount
of data would be to use the LOAD DATA directive. I'm sure most DB's
have similar functionality for loading large amounts of data all at
once. If you are using MySQL read up on LOAD DATA because that is your
best option.


Michael Bleigh wrote: