Mongrels Hanging in mysql_adapter.rb

Hi everyone,

I have been trying to solve this problem for quite some time now and I
really don't have any further clue to investigate.

I am running a mongrel_cluster 1.0.5-2 (mongrel verison : 1.1.5-5) on
a ubuntu box with apache2 2.2.12-1ubuntu2.1

I have 4 different apps on the same box and each app has two mongrel
instances.

After a short period of time ( < 2 hours ) some of the 8 mongrel
instances just stop responding.

I read the mongrel FAQ and tried to set
ActiveRecord::Base.verification_timeout = 14400 (the mysql parameter
is set to the default 28800) in myapp/config/environments/
production.rb

The problem kept happening.

I decided to lower that number, making it 60.
No more success.

I installed gdb and the ruby macros to make sure i was in the mysql
driver while hanging and sure enough, here is the top of the BT from
GDB :

/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/mysql_adapter.rb:264:in `active?'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract_adapter.rb:136:in `verify!'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:256:in
`checkout_and_verify'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:252:in
`checkout_existing_connection'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:186:in `checkout'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:184:in `loop'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:184:in `checkout'",
"/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:183:in `checkout'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:98:in `connection'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:326:in
`retrieve_connection'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_specification.rb:121:in
`retrieve_connection'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_specification.rb:113:in
`connection'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
2934:in `quoted_table_name'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
1626:in `construct_finder_sql'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
1490:in `find_every'",
"/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
589:in `find'",

I figured that since I was running a packaged version of
mongrel_cluster (not the gem installed one), I should use the package
version of mysql driver for ruby.
So i did
gem uninstall mysql

and installed :
apt-get install libmysql-ruby1.8

I then restarted the mongrels and sure enough after 1h25 one of them
was hanging in the mysql_adapter.rb, showing the same backtrace as
before.

I really am out of clue here and any advice / help would be much
appreciated.

I have to note that I am far from being an ROR guru and maybe the
solution to my problem is simpler than what I have been looking for,
but I just don't know where to look anymore.

Thanks for any advice / answer

Damien

For the record, our problem was sitting with the firewall between the app box and the mysql box.

There seems to be a 4000 seconds timeout on any TCP connection passing through the firewall…

I followed the steps detailed here : https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_Timeout

Making it 900.times and the timeout = 1800

It seems to have done the trick.

I don’t like this solution, but hell, it’s the better of two evils I think.

It’ll do for now but if anyone has another way of seing things, be sure to let me know :wink:

Damien