How much "records" can development env stands?!

rubyonrails-talk
1

I didn't know how to title this question sorry :stuck_out_tongue:

I wrote this application using a MySQL DB and Solr Server (updated via acts_as_solr plugin)

I need to load tons of documents to test it out, the app is a newspaper article collection repository that I upload via SGML corpus.

Now the strange is..that after loading about 30K articles..the app started to act weird: it does not load anymore or it does after a lot of minutes..yesterday I've kept loading..it reached the 56K articles..but now is stuck again...

In testing this on a different machine that I use from remote.. any clue?

(the solr server is perfectly reachable...)

thx, Andrea

2

Quoting adedip <adedip@gmail.com>:

I didn't know how to title this question sorry :stuck_out_tongue:

I wrote this application using a MySQL DB and Solr Server (updated via acts_as_solr plugin)

I need to load tons of documents to test it out, the app is a newspaper article collection repository that I upload via SGML corpus.

Now the strange is..that after loading about 30K articles..the app started to act weird: it does not load anymore or it does after a lot of minutes..yesterday I've kept loading..it reached the 56K articles..but now is stuck again...

In testing this on a different machine that I use from remote.. any clue?

(the solr server is perfectly reachable...)

I have a development system with over 5 million rows in one table. The string field is typically 5-50 characters, the rest is a half dozen integers of different sizes. The production system has over 6 million rows.

That said, ActiveRecord is convenient and memory greedy. For that afore mentioned table, I just use straight SQL (inserts and selects can run to thousands of rows). For the other tables, I continue to use ActiveRecord.

HTH,   Jeffrey

3

Jeffrey L. Taylor wrote:

Quoting adedip <adedip@gmail.com>:

of minutes..yesterday I've kept loading..it reached the 56K articles..but now is stuck again...

In testing this on a different machine that I use from remote.. any clue?

(the solr server is perfectly reachable...)

I have a development system with over 5 million rows in one table. The string field is typically 5-50 characters, the rest is a half dozen integers of different sizes. The production system has over 6 million rows.

That said, ActiveRecord is convenient and memory greedy. For that afore mentioned table, I just use straight SQL (inserts and selects can run to thousands of rows).

Have you tried ar-extensions? This should let you do bulk queries properly.

For the other tables, I continue to use ActiveRecord.

HTH,   Jeffrey

Best,