Scaling/optimizing a slow ruby on rails application.

rubyonrails-talk
#1

Hi
everyone,

My
name is Michael Solovyov and I’m the co-founder & CTO of a
rails-based start-up. We have built a product for other businesses
to help their users adopt their products.

We’ve
built the application, delivered to a few customers and now we’ve hit
scaling and performance issues.

I’d
love your feedback on how we proceed as we try to resolve them:

These are the problems we’ve encountered:

The
main user dashboard takes between 7-15 seconds to load. The maximum
requests per second that we can support are 100-150.

Why we think they’re happening:

We’re using
couchdb with an ORM that has our data modeled relationally. This
causes the ORM to launch an extraordinary amount of http requests to
couchdb to load all our data. The more data and users in the project,
the longer the delay. There’s also potentially inefficient code,
extra loops etc. We’re not using caching to it’s fullest extent.

What we’re using/what we’ve done:

Our architecture:
rails ‘3.2.12’, ruby 1.9.3p374, nginx + unicorn (main server),
elasticsearch (separate server), couchdb (separate server). All on
AWS.

We’ve added some
view level caching where we can get away with stale data.

We’ve tried using
higher tiered/ssd aws servers but it doesn’t look to be our main
bottleneck.

How we plan on completing it:

We’re mostly done
a rewrite to migrate our data and data model to activerecord. We’ve
hit a snag in migrating s3 attachment code built into the couchdb ORM
that we’re using. Following the completion of the migration to
sql/activerecord we are going to do the following in an attempt to
increase performance:

A. Use
percono mysql server.

B.
Run tools such as bullet to analyze performance and where we might
have bad queries/lookups.

C.
Update to latest Rails and Ruby versions.

D.
Look into Puma and paralelization of DB access
calls.

E.
Break apart the application into separate services.

What we don’t know right now:

Whether switching
to Activerecord or any of the above ideas will inherently give us any
performance benefits.

Whether we will
have any other unexpected surprises or loss of functionality due to
switching to ActiveRecord.

How long this all
will take and whether there’s other areas that we need to focus to
get sub 1 second performance that we’re missing.

Do
you all have any feedback/ideas/insights completing projects like
these?

In
particular, I would be curious about:

a)
How would you approach it?

b)
Anything that’s worked for you in the past on suitable projects?

c)
Anything to avoid doing?

d)
Where to find good additional team members & advisors to help us
complete the project?

e)
How long to expect it to take?

Here
are our application rake stats:
http://pastie.org/private/qrxkytk4uur9odndydv5dq

#2

We've
built the application, delivered to a few customers and now we've hit
scaling and performance issues.

2)
Why we think they're happening:

We're using
couchdb with an ORM that has our data modeled relationally. This
causes the ORM to launch an extraordinary amount of http requests to
couchdb to load all our data. The more data and users in the project,
the longer the delay. There's also potentially inefficient code,
extra loops etc. We're not using caching to it's fullest extent.

First off storing relational data in couchdb doesn't sound right - it's not really what it is designed for. I used couchdb on a project once many moons ago and things got a lot easier once we stopped trying to use it as a relational datastore.

It seems to me that your first step is to establish a performance baseline: write some automata le performance tests (I don't mean tests in the sense of rspec - just something that exercises your website in a reasonably realistic way). Use tools such as stack_profiler to find out where the time is really going. Tools such as new relic or rack-mini_profiler can also provide insight into what real users are seeing.

3)
What we're using/what we've done:

Our architecture:
rails '3.2.12', ruby 1.9.3p374, nginx + unicorn (main server),
elasticsearch (separate server), couchdb (separate server). All on
AWS.

In my experience elasticsearch is blazing fast, scales well horizontally and doesn't constrict you in quite the way couchdb does (or at least did - couchdb has almost certainly changed since I last used it). You can also use it as a straight up document oriented datastore - I remember reading about some of the early moving elasticsearch folk replacing couchdb with elasticsearch

We've added some
view level caching where we can get away with stale data.

We've tried using
higher tiered/ssd aws servers but it doesn't look to be our main
bottleneck.

4)
How we plan on completing it:

We're mostly done
a rewrite to migrate our data and data model to activerecord. We've
hit a snag in migrating s3 attachment code built into the couchdb ORM
that we're using. Following the completion of the migration to
sql/activerecord we are going to do the following in an attempt to
increase performance:

A. Use
percono mysql server.

It may be a bit late but I would seriously consider postgres. Its richer set of data types (arrays, json etc) might ease the transition and it has a pretty awesome feature set.

B.
Run tools such as bullet to analyze performance and where we might
have bad queries/lookups.

C.
Update to latest Rails and Ruby versions.

Never hurts - also consider GC tuning parameters.

D.
Look into Puma and paralelization of DB access
calls.

E.
Break apart the application into separate services.

5)
What we don't know right now:

Whether switching
to Activerecord or any of the above ideas will inherently give us any
performance benefits.

Whether we will
have any other unexpected surprises or loss of functionality due to
switching to ActiveRecord.

How long this all
will take and whether there's other areas that we need to focus to
get sub 1 second performance that we're missing.

Do
you all have any feedback/ideas/insights completing projects like
these?

In
particular, I would be curious about:

a)
How would you approach it?

b)
Anything that's worked for you in the past on suitable projects?

c)
Anything to avoid doing?

d)
Where to find good additional team members & advisors to help us
complete the project?

e)
How long to expect it to take?

I think this is largely unanswerable by someone unfamiliar with your app. If you're forcing relational style data into a document oriented datastore like couchdb then storing it in a relational db should make it easier/faster. On the other hand if you do play to couchdb's strengths in other ways then that might make the process more difficult (eg if some of your documents store arbitrary key/pairs of data then that can be a pain in the relational world)

Where possible, break the problem down so that you can prove some of these points early (breaking up into services could be a strategy here, if it makes it easier for you to move part of your data across without having to rewrite everything). Writing a prototype that handles some of your hot paths could also provide some key early validation.

I wouldn't obsess too much about tools like bullet - thinking about your data access patterns before this conversion will probably make the biggest difference (although nothing wrong with using it to mop things up afterwards if you find yourself having to include 27 tables then you should probably rethink your design)

Fred

#3

Hi everyone,

My name is Michael Solovyov and I'm the co-founder & CTO of a
rails-based start-up. We have built a product for other businesses to help
their users adopt their products.

We've built the application, delivered to a few customers and now we've
hit scaling and performance issues.

I'd love your feedback on how we proceed as we try to resolve them:

1) These are the problems we've encountered:

The main user dashboard takes between 7-15 seconds to load. The maximum
requests per second that we can support are 100-150.

2) Why we think they're happening:

We're using couchdb with an ORM that has our data modeled relationally.
This causes the ORM to launch an extraordinary amount of http requests to
couchdb to load all our data. The more data and users in the project, the
longer the delay. There's also potentially inefficient code, extra loops
etc. We're not using caching to it's fullest extent.

3) What we're using/what we've done:

Our architecture: rails '3.2.12', ruby 1.9.3p374, nginx + unicorn (main
server), elasticsearch (separate server), couchdb (separate server). All on
AWS.

one easy way you can improve is to move to a more recent version of ruby.
there's a big improvement between
2.0 and 1.9.3 (and it's already 2.1.2) so you might want to look into that
first since it's easier than improving the
code performance.

#4

Without know the internal workings of your application, it is impossible to give specific answers…but if you are experiencing 15 second delays with only a few users, I would take a hard look at the data storage layer. As F. Cheung mentioned, I don’t think you have your data correctly architected with CouchDB. Consider Postgres as your DB, it is robust and has some great features. With only a few users, I doubt you will see much speed improvement upgrading your Ruby or Rails versions. Your immediate concern should be the data architecture.