Mongrels Hanging in mysql_adapter.rb

Hi everyone,

I have been trying to solve this problem for quite some time now and I really don't have any further clue to investigate.

I am running a mongrel_cluster 1.0.5-2 (mongrel verison : 1.1.5-5) on a ubuntu box with apache2 2.2.12-1ubuntu2.1

I have 4 different apps on the same box and each app has two mongrel instances.

After a short period of time ( < 2 hours ) some of the 8 mongrel instances just stop responding.

I read the mongrel FAQ and tried to set ActiveRecord::Base.verification_timeout = 14400 (the mysql parameter is set to the default 28800) in myapp/config/environments/ production.rb

The problem kept happening.

I decided to lower that number, making it 60. No more success.

I installed gdb and the ruby macros to make sure i was in the mysql driver while hanging and sure enough, here is the top of the BT from GDB :

/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/mysql_adapter.rb:264:in `active?'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract_adapter.rb:136:in `verify!'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:256:in `checkout_and_verify'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:252:in `checkout_existing_connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:186:in `checkout'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:184:in `loop'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:184:in `checkout'", "/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:183:in `checkout'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:98:in `connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:326:in `retrieve_connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_specification.rb:121:in `retrieve_connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_specification.rb:113:in `connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 2934:in `quoted_table_name'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 1626:in `construct_finder_sql'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 1490:in `find_every'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 589:in `find'",

I figured that since I was running a packaged version of mongrel_cluster (not the gem installed one), I should use the package version of mysql driver for ruby. So i did gem uninstall mysql

and installed : apt-get install libmysql-ruby1.8

I then restarted the mongrels and sure enough after 1h25 one of them was hanging in the mysql_adapter.rb, showing the same backtrace as before.

I really am out of clue here and any advice / help would be much appreciated.

I have to note that I am far from being an ROR guru and maybe the solution to my problem is simpler than what I have been looking for, but I just don't know where to look anymore.

Thanks for any advice / answer

Damien

For the record, our problem was sitting with the firewall between the app box and the mysql box.

There seems to be a 4000 seconds timeout on any TCP connection passing through the firewall…

I followed the steps detailed here : https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_Timeout

Making it 900.times and the timeout = 1800

It seems to have done the trick.

I don’t like this solution, but hell, it’s the better of two evils I think.

It’ll do for now but if anyone has another way of seing things, be sure to let me know :wink:

Damien