I am trying to work with generating really large CSV files from
active record. (This is actually an end case test however) I am trying
this with
a set of active records that is 700000 records which is just a test
case that I have, though it is very large. The type of find() below is
supposed to work in pages and not have all active records in memory. I
get an out of memory error (see stack dump below). The print of the
count also never comes out.
I got that info from here in the first part of "nailing down the root
cause"
I am also using faster_csv which is supposed to not use as much memory
from soem other google searches.
I am using jruby 1.6.7 (ruby 1.8.7) and active record 2.3 (which I
assume corresponds to rails 2,.3)
on the call here, attr =
{:conditions=>["cdr_guid_id = :cdr_guid_id", {:cdr_guid_id=>30}]}
def self.export_to_csv(attr)
self.set_client(attr[:client])
# client is our own thing,
# and not part of active record find()
attr.delete(:client)
cnt = 0
if hrec = self.find(:first)
CSV.generate(path) do |ofil|
ofil << hrec.visible_attributes.keys
self.find(:all, attr).each do |rec|
cnt += 1
puts cnt.to_s if cnt % 100 == 0
ofil << rec.visible_attributes.keys.map{|col|
rec.send(col) }
end
end
end
end
Thanks, I have tried the find_each and find_in_batches. I seem to get
a local jump error, yield out of block. Why would that be ?
The approach here is to open the csv for appending after the first
batch, but it doesn't work thus far due to the error.
def self.export_to_csv(attr)
self.set_client(attr[:client])
# client is our own thing,
# and not part of active record find()
attr.delete(:client)
cnt = 0
path = attr[:path]
attr.delete(:path)
if hrec = self.find(:first)
mode = 'w'
self.find_in_batches(attr).each do |recs|
CSV.open(path,mode) do |ofil|
# CSV.open(path, 'w') do |ofil|
# ofil << rec.class.column_names
p attr
ofil << hrec.visible_attributes.keys
cnt += 1
puts cnt.to_s if cnt % 100 == 0
recs.each do |rec|
# ofil << rec.class.column_names.map{|col| rec.send(col) }
ofil << rec.visible_attributes.keys.map{|col|
rec.send(col) }
end
end
mode = 'a'
end
end
end