Help! Why is my ActionCaching that slow?

rubyonrails-talk
1

What does a baseline perf test against an action that doesn't hit the database give you? I'm guessing it'll be around 30 req/s.

Mongrel is fast, but definitely not as as fast as lighttpd when it comes to serving static content. If you're running mongrels behind lighttpd, set it up to serve up the cached pages on behalf of mongrel. This'll speed up your perf and let mongrel handle the generation of other pages on your site.

This can be done multiple ways depending on what version of lighttpd you're using...

2

David Scheffel wrote:

Would you say that Mongrel / Rails is fast enough for a havy loaded business web application? Let's say up to 1.500 req/s. Of course I can cluster a bit and use maybe four servers each with a dual core Opteron. Would you say it could do that?

If you clustered, you would be able to handle lots and lots of requests. There are a number of high-traffic sites out there running mongrel+Rails, so this definitely is scalable. With a caveat -- your Rails code is optimized and not relying on stuff like RMagick, etc...

Well, I need to authenticate users. That's why I thought ActionCaching would be the ideal solution.

Yeah, well now you're stuck. Action-level caching will allow you to authenticate, but mongrel will definitely have to touch the request.

From my own testing on my servers, I've noticed that my setup was able

to serve up a whole lot more than httperf was telling me I could serve up. It's probable my testing methods were out of whack, so you might want to check your httperf args are up to snuff. Here's a really good tutorial on using httperf with mongrel. Unfortunately, I haven't gone through it yet.

* http://mongrel.rubyforge.org/docs/how_many_mongrels.html

Oh, and here's a for-pay screencast about httperf: Online Courses, Learning Paths, and Certifications - Pluralsight. It looks like it'll be useful, but YMMV.

3

Would you say that Mongrel / Rails is fast enough for a havy loaded business web application? Let's say up to 1.500 req/s. Of course I can cluster a bit and use maybe four servers each with a dual core Opteron. Would you say it could do that?

Is that 1500 *pages* per second? Or requests? I just checked three sites (pet project, work site, and rubyonrails) and just loading the homepage resulted in 13, 57, and 21 requests (images, javascript, stylesheets). I know our work is high, so let's say 20 is average... that means you want to handle 75 pages/sec which should be very doable on four servers...

-philip

4

I just checked three

sites (pet project, work site, and rubyonrails) and just loading the homepage resulted in 13, 57, and 21 requests (images, javascript, stylesheets).

A serious public site should have a limit per C Class segment or IP-Address via iptables. So the above sites should be able to serve far more than that.

Sorry, what I meant to say was that a single loading of their homepage resulting in 13, 57, and 21 total requests, not that the site could only support that many requests/sec.

-philip

5

Philip Hallstrom wrote:

Sorry, what I meant to say was that a single loading of their homepage resulting in 13, 57, and 21 total requests, not that the site could only support that many requests/sec.

-philip

Good point! Maybe I missunderstood the "experts" I've talked with. They were talking about 1,500 req/s for a forum / chatting plattform. I was told that they are also using a lot of AJAX and that the bottleneck is a mySQL database. That's why they are now using a cluster. Probably that figure included requests to static content like stylesheets, images, and stuff as well.

That's a busy forum, but depending on how they use AJAX I could see it reaching that.

But I thought most of that is cached by the browser anyway? If the clients are frequent users of that portal, wouldn't the browser avoid loading that stuff and request only the dynamic content?!

Guess it depends... on our site most of the pages include their own CSS and specific images as the content is pretty different from section to section, but yeah, after awhile browsers should cash that.

But my point was that lighttpd/nginx/apache/etc should be able to serve thousands of requests per second for static content. So that really shouldn't factor into things unless you're youtube or flickr :slight_smile:

However I'm trying to avoid having havy database load with a intelligent caching strategy. I hope that mongrel will support me on this way.

I'm not sure how mongrel would help specifically with this. Rails page caching and fragment caching would though. Look into memcache as well and the various plugins that tie memcache into AR's find methods and Rails caching in general. Memcache is a life saver for us.

At work I'm maintaining the source-code of a JSP-based web-application used by about 80.000 active customers (meaning customers that log-in at least once a month or so). I've had a look into Tomcat's log-files and found 40 dynamic requests logged for one second somewhere at the afternoon on one of the two machines.

So maybe 100..200 requests/sec. would already be ok for a, let's say medium sized, web-application. If I'm lucky and the site get's even more traffic, I guess I should still be able to serve 1,000 req. with more server-hardware, more mongrels, and a lot of caching so that the database won't get overloaded?!

Does ANYONE knows some more performance figures of rails based web-sites? I would be interested in how many requests they can serve and what type of hardware and software-setup they are using. Too bad the author of "Agile web development with rails" stopped talking about these figures after the first book.

Sometime last year our corp site did 8,996,175 pages and 63,571,374 requests in one day. That works out to um... about 100 pages/sec and 735 requests/sec.

And while I don't trust alexia exactly, for comparision with some other rails sites...

http://img312.imageshack.us/img312/3569/alexavd9.png

We did this with 20 servers running apache and 4 mongrels each, and three separate media servers (for video). None of the servers were overworked (load < 1) so I imagine we could have gotten by with a lot fewer, but the year before we were PHP and were slammed and our traffic triples about this time every year so we didn't want to take any chances :slight_smile:

-philip

6

Hey, I just wanted to dispel a potential myth here.

Many people feel that a server with load over 1.0 is overworked, but that's not the way the master Unix folks saw it.

A process that's blocking on anything, including I/O, network, etc. (think MySQL query) is "waiting to run" and therefore counts towards the run queue.

The best tuners I've known feel that a load of 4.0 is saturated, and target 2-3 for day-to-day operations, sometimes much higher if there are external dependencies of any sort.

Load numbers in and of themselves don't mean much. Why is the load high? What is causing the load to increase? If it's not a local resource constraint, i.e. CPU or Disk I/O, then it's very easy to entirely misunderstand load as it related to performance.

My point is not to suggest you don't know this, but to make sure nobody takes you statement above and misconstrues its meaning.

7

I'm not sure how mongrel would help specifically with this. Rails page caching and fragment caching would though.

I've forgot that it is rails who does the caching. What would you say? Is mongrel still the option #1 for havy loaded apps? Or would you say fcgi is doing a better job here?!

My personal journey has been fcgi -> mongrel -> litespeed. My understanding is that mongrel is preferred over fcgi and that nginx/mongrel seem to be the current favorite, but search the archives as there are others who have done some research on it :slight_smile:

In speaking of memcache. (How) could content be invalided? On the file-system it's easy to write a .sh-script which starts deleting after the 100,000st file, ordered by last access. It that possible with memcache as well?

Memcache has built in expiration. When you store something into memcache you can tell it when to expire (in seconds).

What raise of performance did you noticed with memcache in your environment?

We'd be dead without it :slight_smile: We use it to generate a lot of aggregate queries as well as fragment cache a lot of content.

the various plugins that tie memcache into AR's find methods and Rails caching in general. Memcache is a life saver for us.

Various plugins? Is there more to recommend than the well known MemCacheStore Ruby interface?

Hrm. I don't know their names off hand, but there are plugins to use memcache for sessions, fragment caching, and hook them into Model.find(...)

Sometime last year our corp site did 8,996,175 pages and 63,571,374 requests in one day. That works out to um... about 100 pages/sec and 735 requests/sec.

All pages served by rails? Or most served by apache?!

With the exception of the homepage which was cached for one minute then regenerated all the pages were served by rails.

We did this with 20 servers running apache and 4 mongrels each, and

Just 4 mongrels per server? Was that the optimum configuration for you?

Yep. Moving to 5 hurt performance. See the mongrel site for details.

-philip

8

Philip Hallstrom wrote:

My personal journey has been fcgi -> mongrel -> litespeed.

How much faster is Ruby LSAPI than mongrel incl. the required load balancer?

That I can't tell you as I haven't done the homework myself. All of our stuff is behind a hardware load balancer though so that part isn't relevant.

Memcache has built in expiration. When you store something into memcache you can tell it when to expire (in seconds).

That won't do it for me. I intend to keep as many pages as possible in the cache. Therefore I'd like to expire only those cache-items which have been rarely requested (ordered by last access-time => ls -h).

Then you want cache sweeping and observers and set something up to wipe the cache only if the data that uses it has been updated. Cause if it hasn't, you might as well let it sit there forever :slight_smile:

-philip

9

Memcached afaik will push older items out of the cache as you put new ones into the cache. So you can just keep stuffing things into memcached and the older unused cached items will get pushed out of memcached when new items come in and it has used all its configured memory.

Cheers- -- Ezra Zygmuntowicz-- Lead Rails Evangelist -- ez@engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)