Webbrick and/or Mongrels Dies Overnight

rubyonrails-talk
1

I have a SuSE 10.0 server that was first running Webbrick, and now Mongrel. In both cases, I can have them running and used all day. However, either one of them will go unresponsive in the middle of the night for seemingly no reason. Being that I only use Mongrel now, I check the mongrel.log, and nothing unusual there. I check /var/log/ messages, and nothing there.

Still, a request to http://myhost:3000/mycontroller/list just times out. So I stop Mongrel, and restart as a daemon like: /usr/local/ruby/bin/mongrel_rails start -d --environment production

...and all is fine until the next morning, where the app again goes unresponsive sometime during the night. In my mongrel.log this is the first line of me stopping Mongrel once it is unresponsive and through to the point where I have to stop/start the next day:

** TERM signal received. ** Daemonized, any open files are closed. Look at log/mongrel.pid and log/mongrel.log for info. ** Starting Mongrel listening at 0.0.0.0:3000 ** Starting Rails with production environment... ** Rails loaded. ** Loading any Rails specific GemPlugins ** Signals ready. TERM => stop. USR2 => restart. INT => stop (no restart). ** Rails signals registered. HUP => reload (without restart). It might not work well. ** Mongrel available at 0.0.0.0:3000 ** Writing PID file to log/mongrel.pid

What else could I look for? I have Mongrel running on a another server too, and it stays up fine. In theory, they are set up the exact same way.

Ideas? Rob

2

There's a daemon specifically designed for this purpose (restarting unresponsive or otherwise dead services), called monit.

   Easy, proactive monitoring of processes, programs, files, directories, filesystems and hosts | Monit

n

Ryan Allen wrote:

3

Thanks for the response, Ryan.

Ya, this app is an internal application, not exposed to the outside world. So I don't think it is a bot exploit (though you never know). There would be a number of security/firewall hoops to jump through for an outside resource to access this app.

I was starting to write a sript to restart Mongrel nightly. But then stopped, thinking, I am fixing the symptom instead of the problem. I was hoping to find the source of the issue. With this same app running successfully, without interuption, on our development box, I don't think is it code releated.

Hmmm....still thinking though. Rob

4

Thanks Nathan. Ya, if I go the route of auto restarting every night, I may use that.

I still hope to find the source of the problem instead of just band- aiding the bleeding though. I am guessing that most folks don't have to restart their Mongrel daemon every night.

Rob

5

Have you checked the potential issues in the Mongrel FAQ?

J.

6

Is Mongrel locking up? Are you using pure Ruby MySQL driver?

There is a known issue with pure Ruby MySQL driver that causes it to lock up after a period of inactivity (what actually happens is that MySQL drops an idle connection, and pure Ruby MySQL driver doesn't handle that situation well.

Another regular reason for Mongrel hangups is logs rotation.

Mongrel FAQ lists a number of other possible causes of this behavior.