massive data output

eskim wrote:

Hi.
I've got huge size of data to show to client browser.
It is actually text rows of database.
It's about 200,000 rows and 10 columns.

When I tried it by save it as instance variables and then pass it to
rhtml,
it made server freeze since it is huge data and needs very big size of
memory.

I'm wordering if there is any other way to accomplish this.
I think FLUSHING data output depending on data size.

I've had a similar problem when manipulating big sets of records
(+1000). It basically keeps all of the ActiveRecord objects around in
memory, and it just blows up.

Maybe some kind of scheme where you retrieve the records in batches (so
that you don't have one big array referencing them all), and then force
garbage collection on them? I don't enough about Ruby internals to know
if this is a good idea or even possible, though.

Or, if you're just doing a kind of data dump, perhaps you could
shortcut part of ActiveRecord, access the database adapter and retrieve
the data directly (somehow). Again, I don't know enough ActiveRecord
off the top of my head to say whether this is easily achievable or
advisable. But it's another idea.

Chris