Migration: upload a large set of data via a separate sql file?

rubyonrails-talk
1

Hi all,

I need to upload a large set of data (the cvs file is about 70MB) into some look up tables. Using standard migration api would be tedious. I know that there's a way to export data into YAML file and populate data using this file. However, I think this is still heavy.

Is there a way to upload data using a separate sql file in migration?

Thanks

2

You can use the approach described here for loading a straight SQL file from migrations: Revolution On Rails: Data Loaders in Migrations

Val

3

This is directly from one of my migrations:

   # Get all that research data loaded!    def self.up      say_with_time("Create Tables and Indexes...") { create_tables_and_indexes }      say "Reset all models"; reset_all_models      say_with_time("Import all data...") { import_data }    end

The import_data method used FasterCSV to load a 1.6Mb CSV file into three tables with ActiveRecord models. The thing to remember is that you can do *anything* in your migrations. If the data were already in an SQL file, use the Revolution Health approach. If your data is in CSV, use FasterCSV. The "migration api" only deals directly with the schema changes.

Do what makes sense in your situation.

-Rob

Rob Biedenharn http://agileconsultingllc.com Rob@AgileConsultingLLC.com