Running a "CSV-row massager" on Fixtures

rubyonrails-talk
1

Hi All,

I was wondering if anyone had run into a similar problem and had advice:

I'm using ActiveRecord (without the rest of Rails) to manage data import for large collections of CSV data files into a database. The problem is that each CSV file comes from a different source, and each source uses completely different conventions for encoding just about everything (Country names, phone numbers, dates, null values, etc)

Is there an easy way to sneak a step into the fixture loading process whereupon I have the chance to normalize the data in each row before it is dumped into the database?

For example, I'd need to turn:

John,Doe,us,4/23/1970,null,null,yes

into:

John,Doe,USA,1970-23-4,1

I know I could just monkeypatch the Rails source, but I was hoping for a cleaner solution.

Thanks, Ted [Singing the data integration blues]

2

Edward wrote:

Hi All,

I was wondering if anyone had run into a similar problem and had advice:

I'm using ActiveRecord (without the rest of Rails) to manage data import for large collections of CSV data files into a database. The problem is that each CSV file comes from a different source, and each source uses completely different conventions for encoding just about everything (Country names, phone numbers, dates, null values, etc)

Is there an easy way to sneak a step into the fixture loading process whereupon I have the chance to normalize the data in each row before it is dumped into the database?

For example, I'd need to turn:

John,Doe,us,4/23/1970,null,null,yes

into:

John,Doe,USA,1970-23-4,1

I know I could just monkeypatch the Rails source, but I was hoping for a cleaner solution.

Thanks, Ted [Singing the data integration blues]

If these are test fixtures why not have a separate script that massages the data beforehand? Otherwise you'll need to do this "translation" everytime you run a test.