production trouble

Hello everybody,

I must say that putting rails in production has been a rather
frustrating experience. It maybe just me of course...

At first I simulated the deployment process on my home computer. I
wrote up deploy.rb, played with it until it worked right, deployed at
least a dozen times locally (into another directory) and finally
decided to deploy to my production machine.

The first thing that happened on the remote machine is that
mongrel_cluster started mongrel instances in the development mode. So,
I had to put environment: production into mongrel_cluster.yml. OK,
that worked fine. So now I had 4 instances of mongrel running in the
production mode. I could connect to each of them on their respective
ports no problem. So far so good.

Then I started configuring Apache. In my setup, Apache listens on 9000
and forwards the requests to a balancer (using the standard issue
proxy balancer setup). The problem is that when I access Apache on
9000, the application starts in the development mode! How could that
be? If I modify environment.rb and force the production mode, it works
OK.

But why the heck is all this defaulting to the development mode?

Thanks for any tips...

Sergei

I've heard of this happening on some shared hosts. The only solution
I've ever heard of is to force the environment.rb into production
mode.

That said, I have to say it -- Use Passenger (modrails) if you have
the access to install it. I've had zero problems since I've converted
my servers to it.

Never heard of Passenger, but I'm eager to look into it. I'm glad I'm
not the only one though. How do I force the environment only on the
remote machine? I mean I don't want to take environment.rb from under
subversion. What's the clean way of doing it?

I would like to see your mongrel_cluster.yml file which needs to be
different from dev to test to production... Its best to create this
file on the fly when deploying through capistrano. (Using an ERB
template )

I'm not sure how you could get to a development rails through apache
if your mongrels was started in production mode... Is the standard
proxy setup proxypass / balancer://mongrels ?

Hello,

Here's my mongrel_setup.yml:

hi,

I would recommend that you add a few more lines but I don't think they
would matter to the issues that you have BUT it would make your setup
more secure...

Yes, they respond individually in production mode, but when you do it
through the balancer I see the development environment. Something is
definitely off somewhere in my configuration. But I have a couple of
ideas to check on.

I'll be back within a few days with an update.

Thanks for the tips!

Ok, I know what's going on but I can't seem to be able to fix it.
What's happening is that Apache is not forwarding requests for the
application to the balancer. And I can't figure out why. Here's my
Apache config (sorry if it's too verbose):

<Proxy balancer://bugtracker>
  BalancerMember http://127.0.0.1:2600
  BalancerMember http://127.0.0.1:2601
  BalancerMember http://127.0.0.1:2602
  BalancerMember http://127.0.0.1:2603
  BalancerMember http://127.0.0.1:2604
</Proxy>

ExtendedStatus On

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

<Location /server-status>
   SetHandler server-status
</Location>

<Location /balancer-manager>
   SetHandler balancer-manager
</Location>

<Directory /var/www/bugtracker/current/public">
    Options FollowSymLinks
    # AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

  RewriteEngine On
  RewriteLog /var/log/apache2/myapp_rewrite_log
  RewriteLogLevel 9 # This has no effect -- nothing is written to the
log for some reason

  RewriteRule .* balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]

  # Rewrite index to check for static
  RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page
  RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f

  RewriteRule ^/bugtracker(.*)$ balancer://bugtracker%{REQUEST_URI}
[P,QSA,L]

  AddOutputFilterByType DEFLATE text/html text/plain text/xml
application/xml application/xhtml+xml text/javascript text/css
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html

  ErrorLog /var/log/apache2/rails_errors.log
  CustomLog /var/log/apache2/rails.log combined
</VirtualHost>/

I took the config from here:
http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you
and modified the appropriate stuff.

Whenever I try something like "links http://localhost:9000", I get
"Forbidden". http://localhost:9000/server-status and
http://localhost:9000/balancer-manager don't work either.

Oh, by the way, with AllowOverride None uncommented, none of the
directives in .htaccess under public were allowed -- as expected I
think. I listened on ports 2600 through 2604 and nothing came in. So
that's how I know for sure that the requests were not forwarded. The
reason why it works at all (even though in the development mode) is
because apache interprets dispatch.fcgi thanks to the DocumentRoot in
the config.

Any ideas?

I forgot to take out the line "RewriteRule .* balancer://bugtracker%{REQUEST_URI}
[P,QSA,L] " -- I was just trying to force everything to the balancer.
The rewrite log is finally working and here's what I get there on
"links http://localhost:9000/tickets"

127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) init rewrite engine with requested uri /
tickets
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (3) applying pattern '.*' to uri '/tickets'
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) rewrite '/tickets' -> 'balancer://bugtracker/
tickets'
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (2) forcing proxy-throughput with balancer://bugtracker/tickets
127.0.0.1 - - [02/Sep/2008:20:37:28 --0400] [localhost/sid#822a768]
[rid#83e37d0/initial] (1) go-ahead with proxy request proxy:balancer://
bugtracker/tickets [OK]

But the browser still show the forbidden page and nothing comes in on
ports 2600 through 2604.

This worked:

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

  RewriteEngine On

  <Proxy balancer://bugtracker>
    BalancerMember http://127.0.0.1:2600
    BalancerMember http://127.0.0.1:2601
    BalancerMember http://127.0.0.1:2602
    BalancerMember http://127.0.0.1:2603
    BalancerMember http://127.0.0.1:2604
  </Proxy>

  # Redirect all non-static requests to thin
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  ProxyPass / balancer://bugtracker/
  ProxyPassReverse / balancer://bugtracker/
  ProxyPreserveHost on

  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  # Custom log file locations
  ErrorLog /var/log/apache2/rails_errors.log
  CustomLog /var/log/apache2/rails.log combined
</VirtualHost>

Found that config here: http://articles.slicehost.com/2008/5/9/ubuntu-hardy-apache-rails-and-mongrels

I was missing these lines of course:

ProxyPass / balancer://bugtracker/
ProxyPassReverse / balancer://bugtracker/
ProxyPreserveHost on

But the example at http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/
didn't have them either... And it did work on my local machine. What
was my mistake precisely you think?

Alright, I got it all working correctly. It was all because of haste.
I made a bunch of errors in the apache config and I paid for it... But
at least I was helped by you guys. Thanks very much! Here's my final
config (no ProxyPass).

<VirtualHost 127.0.0.1:9000>
ServerName bugtracker
DocumentRoot /var/www/bugtracker/current/public

  RewriteEngine On

  <Proxy balancer://bugtracker>
    BalancerMember http://127.0.0.1:2600
    BalancerMember http://127.0.0.1:2601
    BalancerMember http://127.0.0.1:2602
    BalancerMember http://127.0.0.1:2603
    BalancerMember http://127.0.0.1:2604
  </Proxy>

  RewriteEngine On
  RewriteLog /var/log/apache2/myapp_rewrite.log
  RewriteLogLevel 9

  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]

  # Rewrite index to check for static
  RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page
  RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L]

  AddOutputFilterByType DEFLATE text/html text/plain text/xml
application/xml application/xhtml+xml text/javascript text/css
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \\bMSIE !no-gzip !gzip-only-text/html <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  # Custom log file locations
  ErrorLog /var/log/apache2/rails_errors.log
  CustomLog /var/log/apache2/rails.log combined
</VirtualHost>

Notice that in my original config I made a mistake in this line:
RewriteRule ^/(.*)$ balancer://bugtracker%{REQUEST_URI} [P,QSA,L].

It was (and I don't know what made me do that!):
RewriteRule ^/(.bugtracker*)$ balancer://bugtracker%{REQUEST_URI}
[P,QSA,L].

My only question is, why specify "AllowOverride None" at all in the
options for the public directory? Curiously, I got no problems with it
locally, but I had to comment it out when deploying remotely. Sorry
for frequent posting. I'm doing it as I go :slight_smile: