Help! incredible timing differences between dev and production

rubyonrails-talk
1

We have a method that is just taking a ridiculous amount of time in production:

Production: Completed in 84043ms (DB: 35243) | 302 Found [http://x.com/admin/x/ 3262/x] (pid:10052)

Dev: Completed in 268ms (DB: 201) | 302 Found [http://c.local/admin/x/3262/ x] (pid:27599)

To do this test I dumped the production database, loaded it into my dev environment and performed the identical task on the identical data. Our production environment is an Amazon EC2 instance, which ok is not as fast as my MacPro, but still!

What can I do to figure out what is going on here? I assume it is perhaps some sort of db locking? But this happens even when no one else is using the production system.

All ideas and tips welcomed!

2

What can I do to figure out what is going on here? I assume it is perhaps some sort of db locking? But this happens even when no one else is using the production system.

I'd start by looking at the difference between explain statements on your local machine and the EC2 instance. If you can narrow it down to one particular query that is a lot slower then that would certainly help.

Fred

3

What I have done is turned on debugging in a console session on production and then executed the methods. They are blazing fast - at least as fast as on dev! Why would executing them via passenger take so much longer.

It is a very consistent thing - every request to this certain method often times out the browser it is so long. Yet... from console it is fast. What would be different about these two (passenger call vs console on same box)?

4

I'd imagine in the console you are only looking at the model end of the stack. Is the response in the browser have a large volume to return & lots to render? That will easily account for the difference.

5

If you have narrowed it down to one method you could put some logging in there, logging with the current time, and narrow it down further.

Colin

6

Quoting phil <phil@philsmy.com>:

We have a method that is just taking a ridiculous amount of time in production:

Production: Completed in 84043ms (DB: 35243) | 302 Found [http://x.com/admin/x/ 3262/x] (pid:10052)

Dev: Completed in 268ms (DB: 201) | 302 Found [http://c.local/admin/x/3262/ x] (pid:27599)

To do this test I dumped the production database, loaded it into my dev environment and performed the identical task on the identical data. Our production environment is an Amazon EC2 instance, which ok is not as fast as my MacPro, but still!

What size EC2 instance are you using? And what size is MacPro? A small EC2 instance is the equivalent of a 1GHz Opteron. I'd expect it to be slower than a 2.XGHz Quad-core. And are they running the same DB server. And is the DB server in the same instance? My small EC2 instance is slower than my 1.6GHz single-CPU laptop, but only 10-20%. Both are running Apache, Passenger, and MySQL.

I don't find significant differences between running Webrick and Apache w/ Passenger on my laptop in terms of response times.

Jeffrey

7

I think I have found the answer - sweepers! We are using a file_store and when the sweepers run on the ec2 instance it is taking forever. (like 10 seconds to rm a directory). Obviously it is time to revisit the caching!