I have an ROR application thats been running on the production server
for almost a couple of months now. Everything seemed fine but the past
few days have caused worry. The whole server slows down drastically
and comes to a halt suddenly and it requires a reboot for the apps i
have on the server to start being responsive again.
Memory leak?
I installed binarylogic's memory logic GitHub - binarylogic/memorylogic: Adds in proccess id and memory usage in your rails logs, great for tracking down memory leaks
which shows me memory usage on every line of my log along with the
PID. I can see that the memory usage keeps going higher with every
request on my local dev machine.
Does that mean ive got a memory leak? How do i diagnose this? I have
no experience with this and im not even sure what a bad number is when
i see one in terms of memory usage.
For information's sake, when i start the app up locally, mem starts at
around 46700 and keeps going up by about 400 with every request on
average. How can i interpret this data? Anyone used memory logic
before?
That's probably a bad thing. You need to start digging deeper: every
request or only some requests ? Run some tests that hit a certain
action 1000 times running: can you narrow it down to a single action
(or set of actions) ? Can you remove the leak by disabling bits of
functionality or commenting out code ? Keep narrowing it down until
you have your culprit. You may find a tool like bleak house (http://
blog.evanweaver.com/articles/2007/05/06/leak-proof-direct-heap-
instrumentation-for-bleak_house/) is usefull too: these things can
tell you what objects are being leaked which may help you find the
problem too.
Lastly, watch out for things which naturally survive between requests:
class variables etc.
I have a few basic questions that if you could answer, would help me
get a better grasp of things.
1. This number, the memory usage, has to go down after fractionally
increasing when a request is made right? So if my app is consuming 40
MB of memory on startup, after going up a little for every request,
ideally, it should come back down to 40MB.. right? For the record,
thats not happening.. mem usage just keeps going higher and higher..
2. If you have ever used New Relic RPM, you would know the Physical
Mem usage graph in there.. its mem usage against time.. Im guessing it
shold be close to constant? Mine is all over the place.. it spikes, it
breaks, it falls and it levels.. i really dont understand why it
breaks altogether! or any of the others.. your thoughts?
3. The way to go about diagnosing this is overloading every action one
by one? I used Apache Benchmark once to do this to our homepage. Is
that good enough? What should i be looking for when im looking for the
leak?
4. As for class variables, the only ones I use are @@per_page for the
will_paginate gem used for pagination. Besides this, there are a few
in the plugins. my sessions hash carries way too little to be causing
this. Besides, i clear it regularly. Any thoughts?
5. Should i be worried about the number of instances of my app that
ive configured mod_rails to serve? I dont quite remember how many that
was and dunno where to change it. Just vaguely seemed like something
that could be related to the problem at hand.
Hope you can guide me here. I have little to no idea about these
things and could really use an experienced one guiding me.. thanks!
I have a few basic questions that if you could answer, would help me
get a better grasp of things.
1. This number, the memory usage, has to go down after fractionally
increasing when a request is made right? So if my app is consuming 40
MB of memory on startup, after going up a little for every request,
ideally, it should come back down to 40MB.. right? For the record,
thats not happening.. mem usage just keeps going higher and higher..
Ideally yes - this may not happen every request, but that should be
the trend
2. If you have ever used New Relic RPM, you would know the Physical
Mem usage graph in there.. its mem usage against time.. Im guessing it
shold be close to constant? Mine is all over the place.. it spikes, it
breaks, it falls and it levels.. i really dont understand why it
breaks altogether! or any of the others.. your thoughts?
pretty constant on my apps (barring instances restarting and stuff
like that)
3. The way to go about diagnosing this is overloading every action one
by one? I used Apache Benchmark once to do this to our homepage. Is
that good enough? What should i be looking for when im looking for the
leak?
Probably good enough. I'd look at the data from memory logic to see if
you have actions you suspect more than others. Hammer them and if
memory usage rockets then you've got a step closer to identifying your
leak, if not you haven;t
4. As for class variables, the only ones I use are @@per_page for the
will_paginate gem used for pagination. Besides this, there are a few
in the plugins. my sessions hash carries way too little to be causing
this. Besides, i clear it regularly. Any thoughts?
5. Should i be worried about the number of instances of my app that
ive configured mod_rails to serve? I dont quite remember how many that
was and dunno where to change it. Just vaguely seemed like something
that could be related to the problem at hand.
wouldn't think so. There have been the odd memory leak in rails (the
ones I remember were in rails 2.2.2), so not a bad idea to scan
through the changelogs etc. to see if there have been any fixed memory
leaks and whether any of them sounds like they might affect you.
you should check your daemons/ crons/ background tasks if you have any
running. try and stop them and see if things are ok. my problem was
with a couple of daemons i was running. i stopped them and im running
a system cron instead. took care of the hanging..
