Guidelines to developing with scaling in mind

I've been reading a lot about how many larger sites are failing due to
the scaling issues RoR has out of the box. I was looking at the well
known Twitter problems, even that somewhat comical slideshow they put
up.

Perhaps with all the tips that are hinted out there, maybe there's a
way to develop RoR from the beginning with scaling issues in mind,
such that at production time when you're surprisingly successful (we
can all dream right?) you wont' be faced with so many scaling
pitfalls.

Are there any articles / books / tutorials on scaling best practices
for RoR?

I want to continue development on RoR but I'm afraid to with the
scaling issues. We may move over to a PHP based framework like
Symfony or look into the RoR port to PHP Akelos.

Please let me know if it is a smart thing to stick to RoR despite
scaling issues. I know they can be worked out, but not sure if that
can justify it.

I've been reading a lot about how many larger sites are failing due to
the scaling issues RoR has out of the box. I was looking at the well
known Twitter problems, even that somewhat comical slideshow they put
up.

Perhaps with all the tips that are hinted out there, maybe there's a
way to develop RoR from the beginning with scaling issues in mind,
such that at production time when you're surprisingly successful (we
can all dream right?) you wont' be faced with so many scaling
pitfalls.

I was at the session on scaling by the twitter people at railsconf
europe last year, and they basically said 'don't do that' (or in other
words, don't optimise prematurely ).

Fred

I've been reading a lot about how many larger sites are failing due to
the scaling issues RoR has out of the box. I was looking at the well
known Twitter problems, even that somewhat comical slideshow they put
up.

Perhaps with all the tips that are hinted out there, maybe there's a
way to develop RoR from the beginning with scaling issues in mind,
such that at production time when you're surprisingly successful (we
can all dream right?) you wont' be faced with so many scaling
pitfalls.

Are there any articles / books / tutorials on scaling best practices
for RoR?

No. But there are resources out there to on scaling best practices.

If you are going to get as enough traffic that you need to scale you'll need to scale regardless of the underlying platform.

It's not a Rails issue. Rails scales just fine as long as you know and understand the rest of it. Rails is a very very small piece of that puzzle.

That said, if you think you're going to be scaling think now about every single piece of your application being on a separate server -- or multiple servers. That will make you think about things like cookies, how to handle uploaded resources (such as avatars say), etc.

But none of that is a Rails issue. You'd have to figure the same thing out for every other framework out there.

For the record, I worked on a Rails site where we didn't do a lot of page caching (we cached db calls quite a bit though) and did over 8 million *pages* a day (not hits. pages. actual rails requests). That's not the biggest, but if you need a number that is plenty big enough to dispell the myth that Rails has scaling problems.

-philip

Since the scaling question seems to keep coming up, I put together an
article that summarizes what I've been able to collect. I'd appreciate
any feedback on it.

http://www.buildingwebapps.com/articles/13-can-rails-scale-absolutely

Michael Slater
www.BuildingWebApps.com

P.S. Philip, can you identify the site that was doing 8 million pages
a day? I haven't been able to document any Rails sites with that much
traffic. The biggest I've found is yellowpages.com, which
extrapolating from the compete.com figures, does something in the
vicinity of 3-5 million pages a day.

Take a look at:

http://blog.caboo.se/articles/2007/7/29/scale-rails-from-one-box-to-three-four-and-five