Guidance for Testing Apps that require large Data Sets

I’m currently trying to build through TDD an app that scrapes daily statistical sports information and puts it into varying models in the hopes of analyzing and researching the data in my own way. Though, to test a lot of the functionality, I require that the database have many records across many database tables (including one that has about 15 columns of information).

Can anyone point to an easy repeatable way to post data when I run my tests, using RSpec, so that feature testing can easily be run.

Thanks in advance

I’m responding on the phone so I will be brief.

A large data set is concerned with the number of records in your tables, not the number of columns. If you can test with just a few or a single row, then you shouldn’t have to do anything special. If your tests will actually require large number of rows, you maybe want to do something different. However, your tests and close could probably benefit from more unit testing with tiny data sets, or even separating things out into POROS instead of having everything in the data models. In addition to that, if your code will be working on statistics on large number of records, consider writing direct sql to take full advantage of the dbms.