Rails 2.3.3 has a very serious performance problem

On my Ubuntu 8.04 64 bit desktop, I created an empty Rails project and
from another machine, I used
ab -n 10 http://210.77.27.169:3000/
to test the performance:

On my Ubuntu 8.04 64 bit desktop, I created an empty Rails project and
from another machine, I used
ab -n 10http://210.77.27.169:3000/
to test the performance:

That's interesting, I didn't get anything like that difference
locally. in fact rails 2.3.3 was faster.

Fred

When you did this "test" did you run the tests multiple times and take
the average? Was this something that was reproducible?

Were they both running in production mode?

Colin

Yes, I ran it several times. Both on a Windows machine of my colleague
and my Linux box. In fact, it is my colleague complaining about the
low efficiency when testing it locally after I upgrade rails from
2.2.2 to 2.3.3 that drove me to do this test.

No, they all ran in development mode. In our production machines, I
did not notice such huge difference in performance. If it is because
of the development mode, I wonder:
1. What slows Rails 2.3.3 down in development?
2. How to overcome it? We need to test our app in our development
machine, this slows down our development significantly and makes the
developers very frustrated when they need to wait for each request.

Did you visit your app by localhost or by your IP address? Visiting by
localhost is quite fast.

No, they all ran in development mode. In our production machines, I
did not notice such huge difference in performance. If it is because
of the development mode, I wonder:
1. What slows Rails 2.3.3 down in development?
2. How to overcome it? We need to test our app in our development
machine, this slows down our development significantly and makes the
developers very frustrated when they need to wait for each request.

Does your other message implies that the app responds quickly when
accessing it from localhost ? (I was. if that is the discriminating
factor then it is very curious). Also it was my impression that
webrick/mongrel only pass the request to rails if there isn't a file
in public that matches (ie hitting / just returns index.html with
basically no rails interaction)

Fred

Now I create two rails apps one by 2.2.2, the other by 2.3.3, each
created with users scaffolding. Then I access /users/ to touch rails.
Each test is performed several times. Accessing from local machine,
the performance seem to be no much difference; it's both about 40ms
per request. However, accessing from a remote server, there's a
dramatic difference:
================Rails 2.3.3================
hoolai@db:~$ ab -n 5 http://210.77.26.176:3000/users/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $>
apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 210.77.26.176 (be patient).....done

Server Software: WEBrick/1.3.1
Server Hostname: 210.77.26.176
Server Port: 3000

Document Path: /users/
Document Length: 1301 bytes

Concurrency Level: 1
Time taken for tests: 50.448329 seconds
Complete requests: 5
Failed requests: 0
Write errors: 0
Total transferred: 9200 bytes
HTML transferred: 6505 bytes
Requests per second: 0.10 [#/sec] (mean)
Time per request: 10089.666 [ms] (mean)
Time per request: 10089.666 [ms] (mean, across all concurrent
requests)
Transfer rate: 0.16 [Kbytes/sec] received

Connection Times (ms)
              min mean[+/-sd] median max
Connect: 2 5 4.7 6 12
Processing: 10061 10083 22.3 10080 10121
Waiting: 10060 10082 21.9 10078 10119
Total: 10063 10089 26.3 10085 10133

Percentage of the requests served within a certain time (ms)
  50% 10082
  66% 10089
  75% 10089
  80% 10133
  90% 10133
  95% 10133
  98% 10133
  99% 10133
100% 10133 (longest request)
=====================Rails 2.2.2================
hoolai@db:~$ ab -n 5 http://210.77.26.176:3000/users/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $>
apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 210.77.26.176 (be patient).....done

Server Software: WEBrick/1.3.1
Server Hostname: 210.77.26.176
Server Port: 3000

Document Path: /users/
Document Length: 1298 bytes

Concurrency Level: 1
Time taken for tests: 0.307687 seconds
Complete requests: 5
Failed requests: 0
Write errors: 0
Total transferred: 9140 bytes
HTML transferred: 6490 bytes
Requests per second: 16.25 [#/sec] (mean)
Time per request: 61.537 [ms] (mean)
Time per request: 61.537 [ms] (mean, across all concurrent
requests)
Transfer rate: 26.00 [Kbytes/sec] received

Connection Times (ms)
              min mean[+/-sd] median max
Connect: 2 3 2.2 4 6
Processing: 35 57 46.3 37 140
Waiting: 34 56 46.4 36 139
Total: 37 61 45.9 43 143

Percentage of the requests served within a certain time (ms)
  50% 43
  66% 43
  75% 43
  80% 143
  90% 143
  95% 143
  98% 143
  99% 143
100% 143 (longest request)

When I looked at the console output when Rails 2.3.3 was processing
request, it seemed that each request was responded quite fast. I don't
understand why it took so long for the client to receive the response.
Is it because of rack?

Processing UsersController#index (for 222.35.3.195 at 2009-09-06
09:54:58) [GET]
  User Load (0.2ms) SELECT * FROM `users`
Rendering template within layouts/users
Rendering users/index
  User Columns (1.6ms) SHOW FIELDS FROM `users`
Completed in 21ms (View: 7, DB: 2) | 200 OK [http://210.77.26.176/
users/]