Help! Why is my ActionCaching that slow?

What does a baseline perf test against an action that doesn't hit the
database give you? I'm guessing it'll be around 30 req/s.

Mongrel is fast, but definitely not as as fast as lighttpd when it
comes to serving static content. If you're running mongrels behind
lighttpd, set it up to serve up the cached pages on behalf of
mongrel. This'll speed up your perf and let mongrel handle the
generation of other pages on your site.

This can be done multiple ways depending on what version of lighttpd
you're using...

David Scheffel wrote:

Would you say that Mongrel / Rails is fast enough for a havy loaded
business web application? Let's say up to 1.500 req/s. Of course I can
cluster a bit and use maybe four servers each with a dual core Opteron.
Would you say it could do that?

If you clustered, you would be able to handle lots and lots of
requests. There are a number of high-traffic sites out there running
mongrel+Rails, so this definitely is scalable. With a caveat -- your
Rails code is optimized and not relying on stuff like RMagick, etc...

Well, I need to authenticate users. That's why I thought ActionCaching
would be the ideal solution.

Yeah, well now you're stuck. Action-level caching will allow you to
authenticate, but mongrel will definitely have to touch the request.

From my own testing on my servers, I've noticed that my setup was able

to serve up a whole lot more than httperf was telling me I could serve
up. It's probable my testing methods were out of whack, so you might
want to check your httperf args are up to snuff. Here's a really good
tutorial on using httperf with mongrel. Unfortunately, I haven't gone
through it yet.

* http://mongrel.rubyforge.org/docs/how_many_mongrels.html

Oh, and here's a for-pay screencast about httperf:
http://peepcode.com/products/benchmarking-with-httperf. It looks like
it'll be useful, but YMMV.

Would you say that Mongrel / Rails is fast enough for a havy loaded
business web application? Let's say up to 1.500 req/s. Of course I can
cluster a bit and use maybe four servers each with a dual core Opteron.
Would you say it could do that?

Is that 1500 *pages* per second? Or requests? I just checked three sites (pet project, work site, and rubyonrails) and just loading the homepage resulted in 13, 57, and 21 requests (images, javascript, stylesheets). I know our work is high, so let's say 20 is average... that means you want to handle 75 pages/sec which should be very doable on four servers...

-philip

I just checked three

sites
(pet project, work site, and rubyonrails) and just loading the homepage
resulted in 13, 57, and 21 requests (images, javascript, stylesheets).

A serious public site should have a limit per C Class segment or
IP-Address via iptables. So the above sites should be able to serve far
more than that.

Sorry, what I meant to say was that a single loading of their homepage resulting in 13, 57, and 21 total requests, not that the site could only support that many requests/sec.

-philip

Philip Hallstrom wrote:

Sorry, what I meant to say was that a single loading of their homepage
resulting in 13, 57, and 21 total requests, not that the site could only
support that many requests/sec.

-philip

Good point! Maybe I missunderstood the "experts" I've talked with. They
were talking about 1,500 req/s for a forum / chatting plattform. I was
told that they are also using a lot of AJAX and that the bottleneck is a
mySQL database. That's why they are now using a cluster. Probably that
figure included requests to static content like stylesheets, images, and
stuff as well.

That's a busy forum, but depending on how they use AJAX I could see it reaching that.

But I thought most of that is cached by the browser anyway? If the
clients are frequent users of that portal, wouldn't the browser avoid
loading that stuff and request only the dynamic content?!

Guess it depends... on our site most of the pages include their own CSS and specific images as the content is pretty different from section to section, but yeah, after awhile browsers should cash that.

But my point was that lighttpd/nginx/apache/etc should be able to serve thousands of requests per second for static content. So that really shouldn't factor into things unless you're youtube or flickr :slight_smile:

However I'm trying to avoid having havy database load with a intelligent
caching strategy. I hope that mongrel will support me on this way.

I'm not sure how mongrel would help specifically with this. Rails page caching and fragment caching would though. Look into memcache as well and the various plugins that tie memcache into AR's find methods and Rails caching in general. Memcache is a life saver for us.

At work I'm maintaining the source-code of a JSP-based web-application
used by about 80.000 active customers (meaning customers that log-in at
least once a month or so). I've had a look into Tomcat's log-files and
found 40 dynamic requests logged for one second somewhere at the
afternoon on one of the two machines.

So maybe 100..200 requests/sec. would already be ok for a, let's say
medium sized, web-application. If I'm lucky and the site get's even more
traffic, I guess I should still be able to serve 1,000 req. with more
server-hardware, more mongrels, and a lot of caching so that the
database won't get overloaded?!

Does ANYONE knows some more performance figures of rails based
web-sites? I would be interested in how many requests they can serve and
what type of hardware and software-setup they are using. Too bad the
author of "Agile web development with rails" stopped talking about these
figures after the first book.

Sometime last year our corp site did 8,996,175 pages and 63,571,374 requests in one day. That works out to um... about 100 pages/sec and 735 requests/sec.

And while I don't trust alexia exactly, for comparision with some other rails sites...

http://img312.imageshack.us/img312/3569/alexavd9.png

We did this with 20 servers running apache and 4 mongrels each, and three separate media servers (for video). None of the servers were overworked (load < 1) so I imagine we could have gotten by with a lot fewer, but the year before we were PHP and were slammed and our traffic triples about this time every year so we didn't want to take any chances :slight_smile:

-philip

Hey, I just wanted to dispel a potential myth here.

Many people feel that a server with load over 1.0 is overworked, but that's
not the way the master Unix folks saw it.

A process that's blocking on anything, including I/O, network, etc. (think
MySQL query) is "waiting to run" and therefore counts towards the run queue.

The best tuners I've known feel that a load of 4.0 is saturated, and target
2-3 for day-to-day operations, sometimes much higher if there are external
dependencies of any sort.

Load numbers in and of themselves don't mean much. Why is the load high?
What is causing the load to increase? If it's not a local resource
constraint, i.e. CPU or Disk I/O, then it's very easy to entirely
misunderstand load as it related to performance.

My point is not to suggest you don't know this, but to make sure nobody
takes you statement above and misconstrues its meaning.

I'm not sure how mongrel would help specifically with this. Rails page
caching and fragment caching would though.

I've forgot that it is rails who does the caching. What would you say?
Is mongrel still the option #1 for havy loaded apps? Or would you say
fcgi is doing a better job here?!

My personal journey has been fcgi -> mongrel -> litespeed. My understanding is that mongrel is preferred over fcgi and that nginx/mongrel seem to be the current favorite, but search the archives as there are others who have done some research on it :slight_smile:

In speaking of memcache. (How) could content be invalided? On the
file-system it's easy to write a .sh-script which starts deleting after
the 100,000st file, ordered by last access. It that possible with
memcache as well?

Memcache has built in expiration. When you store something into memcache you can tell it when to expire (in seconds).

What raise of performance did you noticed with memcache in your
environment?

We'd be dead without it :slight_smile: We use it to generate a lot of aggregate queries as well as fragment cache a lot of content.

the various plugins that tie memcache into AR's find methods and Rails
caching in general. Memcache is a life saver for us.

Various plugins? Is there more to recommend than the well known
MemCacheStore Ruby interface?

Hrm. I don't know their names off hand, but there are plugins to use memcache for sessions, fragment caching, and hook them into Model.find(...)

Sometime last year our corp site did 8,996,175 pages and 63,571,374
requests in one day. That works out to um... about 100 pages/sec and
735
requests/sec.

All pages served by rails? Or most served by apache?!

With the exception of the homepage which was cached for one minute then regenerated all the pages were served by rails.

We did this with 20 servers running apache and 4 mongrels each, and

Just 4 mongrels per server? Was that the optimum configuration for you?

Yep. Moving to 5 hurt performance. See the mongrel site for details.

-philip

Philip Hallstrom wrote:

My personal journey has been fcgi -> mongrel -> litespeed.

How much faster is Ruby LSAPI than mongrel incl. the required load
balancer?

That I can't tell you as I haven't done the homework myself. All of our stuff is behind a hardware load balancer though so that part isn't relevant.

Memcache has built in expiration. When you store something into memcache you can tell it when to expire (in seconds).

That won't do it for me. I intend to keep as many pages as possible in
the cache. Therefore I'd like to expire only those cache-items which
have been rarely requested (ordered by last access-time => ls -h).

Then you want cache sweeping and observers and set something up to wipe the cache only if the data that uses it has been updated. Cause if it hasn't, you might as well let it sit there forever :slight_smile:

-philip

Memcached afaik will push older items out of the cache as you put new ones into the cache. So you can just keep stuffing things into memcached and the older unused cached items will get pushed out of memcached when new items come in and it has used all its configured memory.

Cheers-
-- Ezra Zygmuntowicz-- Lead Rails Evangelist
-- ez@engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)