Memory Consumption issues

rubyonrails-talk
#1

Greetings,

I've recently deployed an app that is having some serious memory
consumption issue. Dropping mongrels quite frequently when the
threshold is met.

Initially, we chalked this problem up to RMagick (I know, I know, but we
needed watermarking). We quickly optimized things so all images were
only processed once then neatly stored in their own folder. When an
image is requested, we confirm that it doesn't exist before processing
it. Been tested, and this is working as it should.

According to some memory profiling we've done, there are a couple of key
pages that are creating large amounts of strings that are not being
reclaimed.

Has anyone ever experienced issues similar to this?

Cheers,
DJ

#2

What is memory limit per process?

#3

Dave -

Greetings,

I've recently deployed an app that is having some serious memory
consumption issue. Dropping mongrels quite frequently when the
threshold is met.

Initially, we chalked this problem up to RMagick (I know, I know, but we
needed watermarking). We quickly optimized things so all images were
only processed once then neatly stored in their own folder. When an
image is requested, we confirm that it doesn't exist before processing
it. Been tested, and this is working as it should.

According to some memory profiling we've done, there are a couple of key
pages that are creating large amounts of strings that are not being
reclaimed.

Has anyone ever experienced issues similar to this?

Cheers,
DJ

I too am presently using monit to manage this until I can track down the leak.

Our mongrel's high water mark is 280 mb, and each of the 11 restart a few times per day.

Rmagick is in place, but isolated, and likely not the source.

There's an article somewhere (lost in my booksmarks) that helps track down leaks. As I suspect my leaks are very specific, I have not appealed to the general group, but perhaps this thread will turn out something interesting.

Jodi

#4

Hey Jodi,

I believe our mongrels memory threshold is in the same ballpark. Here
are a couple of articles we've been looking at, following along with
these processes and using the memory analysis tool in the second has
revealed that the strings not getting reclaimed are blank..

http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/
http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails

We've also been using bleak_house:
http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
yet still can't nail down exactly whats going on here.

Jodi, what version of Rails are you using for you project?

Thanks,
Dave

#5

Thanx Dave -

Hey Jodi,

I believe our mongrels memory threshold is in the same ballpark. Here
are a couple of articles we've been looking at, following along with
these processes and using the memory analysis tool in the second has
revealed that the strings not getting reclaimed are blank..

http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/
http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails

We've also been using bleak_house:
http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
yet still can't nail down exactly whats going on here.

Jodi, what version of Rails are you using for you project?

Thanks,
Dave

We're on 1.2.3, mysql 5, ubunutu 6.06 LTS, ruby 1.8.6 (i686).

I'm deploying a new server (throw money at the problem), but will be getting to the leaks next week.

We've had the leaks since day 1, and although we have 1.8million plus pages, controller actions are limited - so I should be able to localize the area quickly. I'll update this thread with anything learned.

thank you again Dave for the references.

Jodi

#6

If RMagick is not the source of leaks currently then this won't help,
but I just do the following:

`convert -resize 480x360 #{temp_file_path} #{file_path}`

My image uploading is low volume, so this works fine for me. I think
it would work well even for much higher volumes, but I haven't
benchmarked the difference between a command line invocation and
RMagick. It's not likely to leak though.

If you track down the problem please post it.

#7

Greetings mem-leakers

Greetings,

I've recently deployed an app that is having some serious memory
consumption issue. Dropping mongrels quite frequently when the
threshold is met.

Initially, we chalked this problem up to RMagick (I know, I know, but we
needed watermarking).

If RMagick is not the source of leaks currently then this won't help,
but I just do the following:

`convert -resize 480x360 #{temp_file_path} #{file_path}`

My image uploading is low volume, so this works fine for me. I think
it would work well even for much higher volumes, but I haven't
benchmarked the difference between a command line invocation and
RMagick. It's not likely to leak though.

If you track down the problem please post it.

I don't have any answers yet, a few clues, but have some tools to suggest:

re: clues. this article looks promising:

http://blog.pluron.com/2008/01/ruby-on-rails-i.html

As for tools, I'm trying out Evan Weaver's bleak-house:

http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html

as it patches ruby, you'll want to use this on your staging server (Evan says he uses the patched ruby in production)

this is the output from Bleak:

   ./homestars/shared/log $ bleak /tmp/bleak.3242.0.dump
632837 total objects
Final heap size 632837 filled, 732169 free
Displaying top 20 most common line/class pairs
  510101 __null__:__null__:__node__
   29450 __null__:__null__:String
    5511 /usr/local/lib/ruby/gems/1.8/gems/memcached-0.8.1/lib/memcached/memcached.rb:295:String
    4692 (eval):1:__node__
    3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:836:String
    3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:836:ActionController::Routing::DividerSegment
    2084 __null__:__null__:Regexp
    1902 /usr/local/lib/ruby/gems/1.8/specifications/tzinfo-0.3.7.gemspec:12:String
    1686 generated_code_(/var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:388):1:__node__
    1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:834:String
    1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:834:ActionController::Routing::StaticSegment
    1287 /var/www/apps/homestars/releases/20080407140541/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/deprecation.rb:92:__node__
    1106 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:827:ActionController::Routing::DynamicSegment
    1056 /usr/local/lib/ruby/gems/1.8/specifications/rmagick-2.2.2.gemspec:13:String
    1051 /usr/local/lib/ruby/1.8/yaml.rb:133:String
     976 /var/www/apps/homestars/releases/20080407140541/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/module/aliasing.rb:28:__node__
     938 /var/www/apps/homestars/releases/20080407140541/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/module/aliasing.rb:29:__node__
     891 __null__:__null__:Class
     845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:945:ActionController::Routing::Route
     845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/actionpack/lib/action_controller/routing.rb:865:Hash

(the above is from a single run of hitting a single url for 15 minutes. working out a test approach presently)

The above is after a 30 minute run - showing 5511 objects lost in memached (funny enough also written by Evan - I've asked him about the leak). The __ leaks are supposedly internal to ruby - boy, that's a load of lost objects -

other than tzinfo (which I've heard is a culprit), it seems to me that routing is generating a load. Anybody have any info on routing and leaks?

I'm at a bit of a loss when digging info ruby leaks/garbage collection. If anybody has a good ref I'd appreciate it.

rails 1.2.6
ruby 1.8.6 (2008-03-03 patchlevel 901) [i686-linux]

thanx.
Jodi

#8

Hey all,

We'll we tracked down our memory issues. We had some complex join
queries going on and it appears that the AR objects created from said
queries were not getting reclaimed.

So if you are still having issues, and have some complex find_by_sql or
find statements using a lot/nested includes. Try breaking those up into
two queries.. Vast improvement here after doing that.

Cheers,
Dave

#9

Thanx Dave -