Hello, folks.
We have a system that is used by all students on the university. Problem is, when the system is overloaded, people keep strinking F5, and eventually things become too slow to be usable (sometimes, even passenger dies).
How to protect against these cases?
Thanks in advance, Mauricio
There's a great screencast series named "Scaling Rails" that may give you ideas.
But yeah, as the previous poster implies, your problem could be anywhere. You may want to try to identify where the main bottleneck is in your application (whether it's the rails server or the db server) and how you can optimize your code or your deployment to reduce the load.
If the system slows down because of the amount of users you will need more horsepower or limit the amount of concurrent users that use the system. In apache you can set the MaxClients setting if you are using a worker MPM. But you need to find where the bottleneck is so that you can fix it.
I like to start with the database first. Tune the database to the max then look at rails then if you still have issues start limiting at Apache or upgrade/add servers.
If it is a problem with your db server reaching its limit, you may be able to optimize your queries to reduce the number required for each request.
Mauricio, As one poster suggested, Google 'Scaling Rails' for a good overview. My first option would be to look for page caching opportunities. Everywhere. Best, BrianP.
If you are using MySQL....check your slow query log http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
This can also help you find slow actions: http://github.com/jtrupiano/slow-actions
Ok, thanks for the help. The problem is, there is no "slow action", it's just the login page, it sends one query to the database (10 records total), then redirects to a list of matters (it's just a "Matter.find(:all)). The problem I'm getting is precisely this: there is nowhere to optimize...
The application is a "Matter chooser" (something like this, sorry, english isn't my primary language): Students login into a page, choose a couple of matters, and then submit a form. Problem is, each matter has a limited number of students, so the first one that submits gets the matter. So, the application stays online for two days, and in the first 3 hours of the first day, there are a lot of students (200+) that tries to login, and when passenger timeouts their connections, keep pressing F5 to update the page... and that is when things get complicated, because everyone that already logged-in simply gets a timeout screen. It's no use trying to optimize anything, because this happens only in the first 3 hours, for example - after this, things go back to normal...
Suggestions? And, somebody said I could limit simultaneous connections in passenger, how do I do this?
Thanks!
Mauricio Szabo wrote:
Ok, thanks for the help. The problem is, there is no "slow action", it's just the login page, it sends one query to the database (10 records total), then redirects to a list of matters (it's just a "Matter.find(:all)). The problem I'm getting is precisely this: there is nowhere to optimize...
How many "matters" are we talking about 10? 1000?.. Are they cached? Why are you using Matter.find(:all)? Can you scope your query to only find the ones that may be of interest for a given student? ilan
About 100 matters. Also, there is no way I can scope the query (but, with about 100 matters, I hardly think this is an issue).
Ilan Berci wrote:
Mauricio Szabo wrote:
About 100 matters. Also, there is no way I can scope the query (but, with about 100 matters, I hardly think this is an issue).
On Oct 11, 8:55�pm, Ilan Berci <rails-mailing-l...@andreas-s.net>
So either keep the matters in cache or load them up on application start and keep them around in memory.. You don't need to get the matters on each request if you believe that is where the bottleneck lies..
ilan
To make life a little clearer, a 'matter' is a "subject" or a "topic" for a course.. The bottleneck is that the students all want their first choice classes so there's a ridiculous amount of connections being made and it's overloading the web server. I think. And I only think that because this happened in my college when they implemented online course selection.
It's exactly the problem! How did they corrected this issue?
Welcome to the pointy end of online course registration.
Cache the initial 'courses' results.
I don't know how your institution has its data organized, but I'll assume that there are X Courses available. Cache either that initial page, or cache the Matter.find() results so you aren't pinging the DB server for redundant information on every request.
I've a feeling that you're stuck with your current design for now, so the solution would be to throw more capacity at the issue inexpensively by changing your web server params or clustering setup to deal with peaks (max clients, DB connections, etc - Scaling Rails reference).
For the future:
Of course, what would work really well for the students would be to show the available seats in any course when the course listing is returned, which means that caching won't help. But that would be a likely big design change.
Another option (too late for this at the current time) is to implement a priority queue for the students. Fourth year students are allowed to register beginning at 9:00am - on the theory that they should be able to get into classes first so they can meet graduation requirements. Third year students are allowed to begin registering starting at 10:00am - they are under the next strictest set of course requirements, second year students at 11:00am, and first year students at 12:00. This forcefully distributes your load a bit, though you'll still see hourly peaks.
Another option is to not show everyone everything. No student is interested in all 100 courses. The odds are that a student has in mind probably 8 courses (hopefully they are smart enough to plan ahead on the chance that they won't get into one of their favorite classes). Put a search form up front so they can enter their 5 (or however many) classes and get just those back. That return set could show the actual available seats in the course at that moment. If a seat is available, the student can try to get a seat in the course by checking Yes or Register or whatever and submitting right away. If it's already filled, the student knows they have to make another selection.
What you'll probably find is that each student searches initially for just those 2 or 3 courses they really truly want, and make the rest of their course decisions based on how those requests work out.
And given that some popular courses fill up fast, I'd encourage you to investigate the concept of a "wait list" for courses. Even though the course is full, a student may want to "wait list" the course just in case someone already in the class changes their mind and drops out (all the students are 'gaming' their schedules, trying to work out days and times). If a seat does become available, then the first person on the wait list should be offered the seat.
Good luck!