Help! incredible timing differences between dev and production

We have a method that is just taking a ridiculous amount of time in
production:

Production:
Completed in 84043ms (DB: 35243) | 302 Found [http://x.com/admin/x/
3262/x] (pid:10052)

Dev:
Completed in 268ms (DB: 201) | 302 Found [http://c.local/admin/x/3262/
x] (pid:27599)

To do this test I dumped the production database, loaded it into my
dev environment and performed the identical task on the identical
data.
Our production environment is an Amazon EC2 instance, which ok is not
as fast as my MacPro, but still!

What can I do to figure out what is going on here? I assume it is
perhaps some sort of db locking? But this happens even when no one
else is using the production system.

All ideas and tips welcomed!

What can I do to figure out what is going on here? I assume it is
perhaps some sort of db locking? But this happens even when no one
else is using the production system.

I'd start by looking at the difference between explain statements on
your local machine and the EC2 instance. If you can narrow it down to
one particular query that is a lot slower then that would certainly
help.

Fred

What I have done is turned on debugging in a console session on
production and then executed the methods. They are blazing fast - at
least as fast as on dev! Why would executing them via passenger take
so much longer.

It is a very consistent thing - every request to this certain method
often times out the browser it is so long. Yet... from console it is
fast. What would be different about these two (passenger call vs
console on same box)?

I'd imagine in the console you are only looking at the model end of the stack. Is the response in the browser have a large volume to return & lots to render? That will easily account for the difference.

If you have narrowed it down to one method you could put some logging
in there, logging with the current time, and narrow it down further.

Colin

Quoting phil <phil@philsmy.com>:

We have a method that is just taking a ridiculous amount of time in
production:

Production:
Completed in 84043ms (DB: 35243) | 302 Found [http://x.com/admin/x/
3262/x] (pid:10052)

Dev:
Completed in 268ms (DB: 201) | 302 Found [http://c.local/admin/x/3262/
x] (pid:27599)

To do this test I dumped the production database, loaded it into my
dev environment and performed the identical task on the identical
data.
Our production environment is an Amazon EC2 instance, which ok is not
as fast as my MacPro, but still!

What size EC2 instance are you using? And what size is MacPro? A small EC2
instance is the equivalent of a 1GHz Opteron. I'd expect it to be slower than
a 2.XGHz Quad-core. And are they running the same DB server. And is the DB
server in the same instance? My small EC2 instance is slower than my 1.6GHz
single-CPU laptop, but only 10-20%. Both are running Apache, Passenger, and
MySQL.

I don't find significant differences between running Webrick and Apache w/
Passenger on my laptop in terms of response times.

Jeffrey

I think I have found the answer - sweepers!
We are using a file_store and when the sweepers run on the ec2
instance it is taking forever. (like 10 seconds to rm a directory).
Obviously it is time to revisit the caching!