production trouble

Hello everybody,

I must say that putting rails in production has been a rather frustrating experience. It maybe just me of course...

At first I simulated the deployment process on my home computer. I wrote up deploy.rb, played with it until it worked right, deployed at least a dozen times locally (into another directory) and finally decided to deploy to my production machine.

The first thing that happened on the remote machine is that mongrel_cluster started mongrel instances in the development mode. So, I had to put environment: production into mongrel_cluster.yml. OK, that worked fine. So now I had 4 instances of mongrel running in the production mode. I could connect to each of them on their respective ports no problem. So far so good.

Then I started configuring Apache. In my setup, Apache listens on 9000 and forwards the requests to a balancer (using the standard issue proxy balancer setup). The problem is that when I access Apache on 9000, the application starts in the development mode! How could that be? If I modify environment.rb and force the production mode, it works OK.

But why the heck is all this defaulting to the development mode?

Thanks for any tips...

Sergei

I've heard of this happening on some shared hosts. The only solution I've ever heard of is to force the environment.rb into production mode.

That said, I have to say it -- Use Passenger (modrails) if you have the access to install it. I've had zero problems since I've converted my servers to it.

Never heard of Passenger, but I'm eager to look into it. I'm glad I'm not the only one though. How do I force the environment only on the remote machine? I mean I don't want to take environment.rb from under subversion. What's the clean way of doing it?

I would like to see your mongrel_cluster.yml file which needs to be different from dev to test to production... Its best to create this file on the fly when deploying through capistrano. (Using an ERB template )

I'm not sure how you could get to a development rails through apache if your mongrels was started in production mode... Is the standard proxy setup proxypass / balancer://mongrels ?

Hello,

Here's my mongrel_setup.yml:

hi,

I would recommend that you add a few more lines but I don't think they would matter to the issues that you have BUT it would make your setup more secure...

Yes, they respond individually in production mode, but when you do it through the balancer I see the development environment. Something is definitely off somewhere in my configuration. But I have a couple of ideas to check on.

I'll be back within a few days with an update.

Thanks for the tips!

Ok, I know what's going on but I can't seem to be able to fix it. What's happening is that Apache is not forwarding requests for the application to the balancer. And I can't figure out why. Here's my Apache config (sorry if it's too verbose):

<Proxy balancer://bugtracker>   BalancerMember http://127.0.0.1:2600   BalancerMember http://127.0.0.1:2601   BalancerMember http://127.0.0.1:2602   BalancerMember http://127.0.0.1:2603   BalancerMember http://127.0.0.1:2604 </Proxy>

ExtendedStatus On

<VirtualHost 127.0.0.1:9000> ServerName bugtracker DocumentRoot /var/www/bugtracker/current/public

<Location /server-status>    SetHandler server-status </Location>

<Location /balancer-manager>    SetHandler balancer-manager </Location>

<Directory /var/www/bugtracker/current/public">     Options FollowSymLinks     # AllowOverride None     Order allow,deny     Allow from all   </Directory>

  RewriteEngine On   RewriteLog /var/log/apache2/myapp_rewrite_log   RewriteLogLevel 9 # This has no effect -- nothing is written to the log for some reason

  RewriteRule .* balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f   RewriteCond %{SCRIPT_FILENAME} !maintenance.html   RewriteRule ^.*$ /system/maintenance.html [L]

  # Rewrite index to check for static   RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page   RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster   RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f

  RewriteRule ^/bugtracker(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css   BrowserMatch ^Mozilla/4 gzip-only-text/html   BrowserMatch ^Mozilla/4\.0[678] no-gzip   BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html

  ErrorLog /var/log/apache2/rails_errors.log   CustomLog /var/log/apache2/rails.log combined </VirtualHost>/

I took the config from here: http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you and modified the appropriate stuff.

Whenever I try something like "links http://localhost:9000", I get "Forbidden". http://localhost:9000/server-status and http://localhost:9000/balancer-manager don't work either.

Oh, by the way, with AllowOverride None uncommented, none of the directives in .htaccess under public were allowed -- as expected I think. I listened on ports 2600 through 2604 and nothing came in. So that's how I know for sure that the requests were not forwarded. The reason why it works at all (even though in the development mode) is because apache interprets dispatch.fcgi thanks to the DocumentRoot in the config.

Any ideas?

I forgot to take out the line "RewriteRule .* balancer://bugtracker%{REQUEST_URI} [P,QSA,L] " -- I was just trying to force everything to the balancer. The rewrite log is finally working and here's what I get there on "links http://localhost:9000/tickets&quot;

127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768] [rid#83e37d0/initial] (2) init rewrite engine with requested uri / tickets 127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768] [rid#83e37d0/initial] (3) applying pattern '.*' to uri '/tickets' 127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768] [rid#83e37d0/initial] (2) rewrite '/tickets' -> 'balancer://bugtracker/ tickets' 127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768] [rid#83e37d0/initial] (2) forcing proxy-throughput with balancer://bugtracker/tickets 127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768] [rid#83e37d0/initial] (1) go-ahead with proxy request proxy:balancer:// bugtracker/tickets [OK]

But the browser still show the forbidden page and nothing comes in on ports 2600 through 2604.

This worked:

<VirtualHost 127.0.0.1:9000> ServerName bugtracker DocumentRoot /var/www/bugtracker/current/public

  RewriteEngine On

  <Proxy balancer://bugtracker>     BalancerMember http://127.0.0.1:2600     BalancerMember http://127.0.0.1:2601     BalancerMember http://127.0.0.1:2602     BalancerMember http://127.0.0.1:2603     BalancerMember http://127.0.0.1:2604   </Proxy>

  # Redirect all non-static requests to thin   RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f   RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  ProxyPass / balancer://bugtracker/   ProxyPassReverse / balancer://bugtracker/   ProxyPreserveHost on

  <Proxy *>     Order deny,allow     Allow from all   </Proxy>

  # Custom log file locations   ErrorLog /var/log/apache2/rails_errors.log   CustomLog /var/log/apache2/rails.log combined </VirtualHost>

Found that config here: http://articles.slicehost.com/2008/5/9/ubuntu-hardy-apache-rails-and-mongrels

I was missing these lines of course:

ProxyPass / balancer://bugtracker/ ProxyPassReverse / balancer://bugtracker/ ProxyPreserveHost on

But the example at http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/ didn't have them either... And it did work on my local machine. What was my mistake precisely you think?

Alright, I got it all working correctly. It was all because of haste. I made a bunch of errors in the apache config and I paid for it... But at least I was helped by you guys. Thanks very much! Here's my final config (no ProxyPass).

<VirtualHost 127.0.0.1:9000> ServerName bugtracker DocumentRoot /var/www/bugtracker/current/public

  RewriteEngine On

  <Proxy balancer://bugtracker>     BalancerMember http://127.0.0.1:2600     BalancerMember http://127.0.0.1:2601     BalancerMember http://127.0.0.1:2602     BalancerMember http://127.0.0.1:2603     BalancerMember http://127.0.0.1:2604   </Proxy>

  RewriteEngine On   RewriteLog /var/log/apache2/myapp_rewrite.log   RewriteLogLevel 9

  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f   RewriteCond %{SCRIPT_FILENAME} !maintenance.html   RewriteRule ^.*$ /system/maintenance.html [L]

  # Rewrite index to check for static   RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page   RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster   RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f   RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css   BrowserMatch ^Mozilla/4 gzip-only-text/html   BrowserMatch ^Mozilla/4\.0[678] no-gzip   BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html <Proxy *>     Order deny,allow     Allow from all   </Proxy>

  # Custom log file locations   ErrorLog /var/log/apache2/rails_errors.log   CustomLog /var/log/apache2/rails.log combined </VirtualHost>

Notice that in my original config I made a mistake in this line: RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L].

It was (and I don't know what made me do that!): RewriteRule ^/(.bugtracker*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L].

My only question is, why specify "AllowOverride None" at all in the options for the public directory? Curiously, I got no problems with it locally, but I had to comment it out when deploying remotely. Sorry for frequent posting. I'm doing it as I go :slight_smile: