weird random errors when running without session affinity (behind balancing proxy)

Hi,

I use nginx serving 2 applications. Each is powered by cluster of at least 2 mongrels. While one application is working fine and running for days without problem, another one is just screwed up *when* running in cluster.

That is, this problematic application runs fine when it is handled by one mongrel. But when it is served by two or more mongrels (with nginx as balancing proxy at front) there are totally strange errors. So far I observed three of them happening very randomly:

- Application Error - caused by not founding controller's class for some unknown reason. And that class is there and it is found by subsequent request.

- 404 Not Found - in log there's Recognition Failed Error for completely valid request. Hitting refresh makes it suddenly working (not everytime).

- Infinite loop - application just go into looping, eating all CPU and doing nothing. No log entry for this.

I think it is because nginx cannot provide balancing with sticky session to backend. So may be storing a normal object (derived from Object) into session causing these weird problems?

Hi Kamil,

How are you storing the session information from within Ruby. If you don't have session affinity then there are configuration options in rubyonrails to store the session data in a database, in a shared (between the servers) file directory or on another server such as memcached (I think on the last one) which would mean you don't necessarily need session affinity.

If session affinity is not working with nginx (which I know nothing about btw) then you will get very strange behavour from the servers as sessions go to the wrong ones.

Looking at the server logs may tell you if the sessions are going to the wrong place. You could also add a filter that would check that the session info was wrong or not found so at least you would know. Just check the header for the presence of the session cookie.

regards

Sean

Hi~

Hi,

I use nginx serving 2 applications. Each is powered by cluster of at least 2 mongrels. While one application is working fine and running for days without problem, another one is just screwed up *when* running in cluster.

That is, this problematic application runs fine when it is handled by one mongrel. But when it is served by two or more mongrels (with nginx as balancing proxy at front) there are totally strange errors. So far I observed three of them happening very randomly:

- Application Error - caused by not founding controller's class for some unknown reason. And that class is there and it is found by subsequent request.

- 404 Not Found - in log there's Recognition Failed Error for completely valid request. Hitting refresh makes it suddenly working (not everytime).

- Infinite loop - application just go into looping, eating all CPU and doing nothing. No log entry for this.

I think it is because nginx cannot provide balancing with sticky session to backend. So may be storing a normal object (derived from Object) into session causing these weird problems?

  What session container are you using? Are you suing just the pstore disk sessions? If so switch to the ActiveRecord store and that may fix your problems. I run very many clusters of mongrels behind nginx with the AR session store and I have none of these issues.

-- Ezra Zygmuntowicz-- Lead Rails Evangelist -- ez@engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)