rails/ruby memory leak

hi all

i've been chasing my tail on a tiny memory leak for a few days. i
isolated the cause to what amounts to a rails project with database
sessions enabled. now i'm not sure that the leak is in rails, in fact
i think it's in ruby's cgi/session.rb but i want to gather a few data
points to confirm the leak i'm seeing. if you have a little time,
check out

  http://drawohara.tumblr.com/post/13655303

or just grab the project from

  http://s3.amazonaws.com/drawohara.com.downloads/leak.tgz

and get it running on your box. it should be *very* quick to do if
you have sqlite3 installed. running it is also easy, but seeing the
leak can take a bit of time - 30 minutes to 6 hours depending on how
fast your box can hammer ./script/server. anyhow, people can report
here or to me directly. i'll summarize my findings and, if there is
in fact a leak in either ruby or rails - follow up with a bug report.

kind regards.

У Няд, 30/09/2007 у 15:59 +0000, ara.t.howard піша:

hi all

i've been chasing my tail on a tiny memory leak for a few days. i
isolated the cause to what amounts to a rails project with database
sessions enabled. now i'm not sure that the leak is in rails, in fact
i think it's in ruby's cgi/session.rb but i want to gather a few data
points to confirm the leak i'm seeing. if you have a little time,
check out

About a month ago I've found similar (or, probably, same) leak while
checking our application for leaks. It leaks very slow. And it's seems
to be not Rails issue. Something with $1, $2, ... variable handling in
ruby itself. I think I still have those valgrind report files at home.

I had not enough time to dig it seriously deep and code in that place of
MRI is not very easy, so I gave up.

that is good to know. everyone who has reported so far has seen the same results: a very slow leak over time. of course it only shows up with sessions, but that is going to affect just about every rails project. at this point i'm more concerned with finding a workaround in rails than trying to patch ruby - but i'm not sure if it's possible. i'm running out of steam too. nevertheless this is a pretty steady leak - it should be fixable.

cheers.

a @ http://drawohara.com/

About a month ago I've found similar (or, probably, same) leak while
checking our application for leaks. It leaks very slow. And it's seems
to be not Rails issue. Something with $1, $2, ... variable handling in
ruby itself. I think I still have those valgrind report files at home.

I had not enough time to dig it seriously deep and code in that place of
MRI is not very easy, so I gave up.

i had forgotten to mention: i do NOT see the leak using FileStore
sessions - only when db sessions are turned on do i see the leak. my
whole app does nothing but render a single string so sessions have
been the only variable. did you notice this? how bare bones of an
app did you test and see this behavior? i've gotten in about 10
reports now and everyone is seeing this slow memory consumption: at
this point i still think it's in ruby but the fact that only database
sessions seem to trigger it has me worried.

cheers.

Just a thought : I do not use sessions in most of my critical applications which need to serve more than 10K req/sec (a magic no. I have zeroed upon). I rely on a completely cookie based authentication system which AFAIK is quiet safe(a lot of central auth system use such an approach), it gives me a major performance boost http://pranjan.blogspot.com/2007/07/wanna-make-your-rails-app-faster.html
Although I do need to hack a few things to show flash and/or error messages as sessions are switched off but I feel it is worth it.

Or of course use the new cookie-base session store.

Fred

that's very interesting indeed. i'm just about do confirm that the
normal file based sessions do not leak and also test stephan kaas'
alternate db backed sessions. if both do not leak i can narrow my
search to rails' activerecord store. if either do i'll be 99% sure
the leak is in ruby and probably have to resort to dmalloc and co.
uggh.

your suggestion, however, is a very good one and could be quite
suitable for our application. thanks a bunch.

that's very interesting indeed. i'm just about do confirm that the
normal file based sessions do not leak and also test stephan kaas'
alternate db backed sessions. if both do not leak i can narrow my
search to rails' activerecord store. if either do i'll be 99% sure
the leak is in ruby and probably have to resort to dmalloc and co.
uggh.

so far, after > 100,000 requests neither FileStore or SQLSessionStore
are not leaking. only ActiveRecordStore is...

so far, after > 100,000 requests neither FileStore or SQLSessionStore
are not leaking. only ActiveRecordStore is...

What rev are you testing? Does it include my new attributes code?
That broke ARStore initially...

What rev are you testing?

cfp:~ > rails --version
Rails 1.2.3

cfp:~ > ruby --version
ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.9.1]

cfp:~ > uname -srm
Darwin 8.10.1 i386

cfp:~ > gem list sqlite3

*** LOCAL GEMS ***

sqlite3-ruby (1.2.1)
    SQLite3/Ruby is a module to allow Ruby scripts to interface with a
    SQLite3 database.

think that's all that's relevant.... (btw i see the leak on mysql and
postgresql too). more info, including test app, here:

  http://drawohara.tumblr.com/post/13655303

Does it include my new attributes code?
That broke ARStore initially...

can you give me a quick grep to tell?

cheers.

ran for 300,000 requests with 1.2.3 and FIleStore - no leak.

check checked out edge and am testing the new cookie based sessions,
then i'll test the database store with edge...

What rev are you testing? Does it include my new attributes code?
That broke ARStore initially...

testing against svn rails shows file and cookie stores not leaking but
activerecord store continuing to leak.

testing against svn rails shows file and cookie stores not leaking but
activerecord store continuing to leak.

So it's not my new attribute code.... I did enjoy the benchmarks you
published though ...

I have another report of a memory leak of some sort which I'm
currently investigating, if I can track it down I'll let you know...

testing against svn rails shows file and cookie stores not leaking but
activerecord store continuing to leak.

So it's not my new attribute code.... I did enjoy the benchmarks you
published though ...

yeah it looks like we can't blame you :wink:

i'm not sure where to go from here - dike (leak detector) keeps suggesting that it's somehow cgi/session.rb:165 that's leaking with arstore but i made zero progress hacking around and above that code to eliminate the leak, also it's odd that file store also uses cgi/session.rb yet does not leak... so, at this point, my gut says it's just something peculiar about the way in which the session lib is used from ar store or, just as likely, some obscure leak in ruby itself that's causing the problem. however, knowing that i can drop cookie sessions in, eliminate the leak, and get a huge speed boost - is mighty powerful motivation not to fix ar store :wink: i'll probably give one more look at debugging in in edge and then cry 'uncle'.

I have another report of a memory leak of some sort which I'm
currently investigating, if I can track it down I'll let you know...

have you checked out dike?

   http://codeforpeople.com/lib/ruby/dike/dike-0.0.3/README

   gem install dike

i'd be curious to know if it helped you any.

cheers.

a @ http://drawohara.com/

Hey Ara, I noticed you're running 1.8.6 patchlevel 0. Could you try
with the current 1.8.6?

I hammered your test app and see rsize solidly between 20428-20532 and
object count consistently alternating between 144773 and 38070 +- 3.

I'm on ruby 1.8.6 (2007-05-30 patchlevel 5000) [i686-darwin8.9.1]

Best,
jeremy

Hey Ara, I noticed you're running 1.8.6 patchlevel 0. Could you try
with the current 1.8.6?

will do.

I hammered your test app and see rsize solidly between 20428-20532 and
object count consistently alternating between 144773 and 38070 +- 3.

for how many requests? for me it takes a good 50-100k to definitively see the leak - as amazing as that sounds...

I'm on ruby 1.8.6 (2007-05-30 patchlevel 5000) [i686-darwin8.9.1]

i'll report back tomorrow or whenever i reach 100k requests.

cheers.

a @ http://drawohara.com/

I stopped at just over 25000 requests to give my laptop fan a rest, poor thing.

I'll try again and let it run longer.

jeremy

So I’m curious then, how can we look at a sessionless based system that still provides functionality like carts and such for our apps?

Anyone reference a blog post or two I can learn from?

stable-snapshot is @ 110

where's that 5k come from?

kind regards.

a @ http://codeforpeople.com/

I built it from the ruby_1_8 branch; that's the default patchlevel.

For example, I just recompiled and now have
  ruby 1.8.6 (2007-10-02 patchlevel 5000) [i686-darwin8.10.1]

jeremy