Article: High-Performance Ruby On Rails Setups Test...

Alexey Kovyrin wrote:

This week we have started one new project with Ruby on Rails as primary framework. My first task was to prepare runtime environment for it on one of our development servers. When I have tried to research how people doing it, I noted that there is no information about how to deploy rails application with nginx as frontend and what is performance of such solution. Before blindly make any decisions about future platform I’ve decided to make some performance tests of the some most popular rails web servers/frontends. Results of these tests you can find here with configuration samples for all software that I have used.

Details: http://blog.kovyrin.net/2006/08/22/high-performance-rails-nginx-lighttpd-mongrel
  

First of all - great initiative. But - as is posted in the comments on your blog - testing the default rails index.html page really isn't performance testing rails at all - since that's a static html page. You need to create an app that probably hits a database on every page view or at least hits and then caches.

I'm also questioning how you came up with testing 100 simultaneous connections from one single computer. I'm sure the tcp/ip stack of the OS in question will be a factor here - and 100 connections simultaneously is a LOT.

As they've already told you, there's a need to make the same tests with
some kind of DB query... Here are my results:

You also have to run each test multiple times and take the mean of them
in order to rule out accidentally getting a bad read. Not to mention a
bunch of other stuff.

<modified below>

Webrick (only T1)
T1: - Req/s: 1.09

Lighttpd + 5 fcgi spawns:
T1: - Req/s: 2.41
T2: - Req/s: 2.31
T3: - Req/s: 2.32

Mongrel (only T1)
T1: - Req/s: 0.92

Pound + 5xMongrel Cluster
T1: - Req/s: 1.72
T2: - Req/s: 1.54

All of these are so pathetically low that I seriously doubt your test is
configured correctly. These are so low that you're either running this
against a worst case setup or in development mode or something. In
fact, these numbers are so low it doesn't matter how fast your setup is,
it can't service any kind of reasonable requests rate.

In your situation there's probably so many confounding factors that
you'd need to work on the setup to find the simplest best case as a
comparison (like the control in an experiment). Until you do that you
really have no idea how fast each config could be.

For example, how is it possible that you have such a low performance and
yet with 5 mongrels or fcgi processes you don't see a 5x increase?
That's the expected response. Also, why 5? How come there's no test
for lighttp+1 fcgi? Is this with everything running on the same machine
(even your ab run)? How many of these runs did you do? Just one? How
do you know that one wasn't just bad/good luck? Why are you increasing
the concurrency but not the number of connections?

Just some minor criticisms as I don't think the test is right for any of
the tested servers (I *know* fcgi does better than 2.3 req/s, as well as
mongrel).

Yes I read it, I was just giving you some friendly advice on how to do
it again so that it actually showed something. Your worst case scenario
is basically making it so that your performance measurement has no
information. You've confounded the analysis with all the extra
conditions you put on it, when it's not really necessary.

For example, how do I know your "worst case scenario" isn't really:

  def sleep_for_days
    sleep 60
  end

I don't, so without a baseline to compare with on the same machine I
have nothing to go on (unless you post your code). What if you run your
baseline and come up with say 20 req/sec? Well, I'd assume your system
is setup poorly since I can get much smaller machines to do better than
that. Without a baseline, I have nothing to compare against.

Now I understand it was done quickly, but if you post measurements like
that without taking the time to do them right then it just gives
everyone a bunch of FUD to deal with. A better approach would be to
have done all your analysis for the baseline fastest case (as a control)
*then* your analysis.

Also, don't just say X is faster than Y. Really clarify it, "With a
blah controller that has a blah condition with blah database and no
caching on foo machine, then X is faster than Y."

Because, just like the nightly news, all people will post on their blogs
is:

  "X is faster than Y".

Which does us all a disservice in the end.