Problem running migration

rubyonrails-talk
1

Hi, I'm trying to run a set of migration which brings data from a legacy project. The legacy project is on production and have lots of data. So, during the time the migration keep allocating and allocating memory until it grabs it all. I'm doing pagination to bring the data step by step. Beside, what is very wear is that the memory never goes down between migration, for example, I do:

rake db:migrate

== 2 InitialMoveLegacyUsersToUsers: migrating

2

Rafael Chacon wrote:

rake db:migrate

== 2 InitialMoveLegacyUsersToUsers: migrating

== 2 InitialMoveLegacyUsersToUsers: migrated (83.3700s)

== 3 AddPasswordResetCodeToUsers: migrating

-- add_column(:users, :password_reset_code, :string)    -> 0.0877s == 3 AddPasswordResetCodeToUsers: migrated (0.0883s)

== 4 AddOpenIdAuthenticationTables: migrating

-- create_table(:open_id_authentication_associations, {:force=>true})    -> 0.0054s -- create_table(:open_id_authentication_nonces, {:force=>true})    -> 0.0044s == 4 AddOpenIdAuthenticationTables: migrated (0.0105s)

And watching the behavior of the memory between migrations it never free memory, always keep growing.

You probably need to show me what your migrations look like. You must be loading a load of database rows into ruby objects?

Maybe you're setting them to a variable that remains in scope?

Don't know.

If you do rake db:migrate VERSION=2 and then seperately do rake db:migrate VERSION=4

does it isolate the problem to the first migration?

3

The most conlficting migration looks like this:

class MoveLegacySugarEntries < ActiveRecord::Migration   class Entries < ActiveRecord::Base;end

  class Entry::SugarEntry < Entry;end   def self.up    Legacy::SugarReading.each do |le|       u = User.find_by_login(le.user.login)       unless u== nil         TzTime.zone = u.tz          Entry::SugarEntry.create!(         :registered_at => DateTime.parse("#{le.date} #{le.time.hour}:#{le.time.min}"),         :comment => le.comment,         :value => le.reading,         :user => u)       end     end   end

Where the active record each is implemented like this:

class <<ActiveRecord::Base   def each(limit=1000)     rows = find(:all, :conditions => ["id > ?", 0], :limit => limit)     while rows.any?       rows.each { |record| yield record }       rows = find(:all, :conditions => ["id > ?", rows.last.id], :limit => limit)     end     self   end end

So, is getting 1000 rows at a time... Enough to don't exploit the memory. Which variable might not be getting out of scope??

Anyway, how would you think possible to get garbage memory from previous migrations?? For example, if I ran my migration from 0 when I got to this migration I already have 1.5 gigas memory occupied, isn't kind of wear???

Thanks for your help