Best Practices; Deploying Updates

Hello,

So I am a running a few shopping carts using spree (open source rails cart) and some of my clients site traffic is starting to rise. Normally I do deploy updates late at night but some times I have to deploy the asap, while customers are still on the site in the process of ordering. Obviously when I deploy updates it takes the site offline for about 15 seconds, which will cause all customers to leave and never return. Is there a way to deploy an update without having the site go offline?

I am running nginx and thin, and I deploy with just a simple bash script. The only thing I was thinking of doing is to have multiple instances of the app running and shut down each instance one at a time, would that work?

Any advice in the right direction would be greatly appreciated

Look into capistrano or one of the other deploy systems out there. Capistrano will deploy the new code into a new directory (while your site stays up), then move a symlink, and restart the app. So downtime is very minimal. It's not perfect in that if you run migrations against the old code weird things could happen, but it's still very nice.

You may also want to look into Unicorn, which can give you no-downtime deploys.

It can if your load balancer is smart enough to notice that is trying to send a request to a dead thin instance and instead send it to another one. (haproxy handles this pretty nicely, or something like http://www.linuxvirtualserver.org/ to balance between several servers).

you take your first server (or first set of thin instances) down, upgrade them and then do the same on the remaining server/instances.

This is not always trouble free - for example your new code might populate tables with data that the old instances still running don't understand, a migration might add a column that the old code won't populate properly etc - I typically only use this to roll out small fixes outside of our normal maintenance period.

Fred

Regardless of the technology, this is what I would do:

Post maintenance notices on the site upfront. This way the customers know what to expect and will not be disappointed if they try accessing the site, while you are updating it, and if they encounter errors.

Communicating and talking to the customers helps.

If you were going to take only 15 seconds, say it will take 5 minutes. When done, remove the update notice and say, "It is back. Thank you for your patience" or something on those lines.

Surya Avantsa

suryava wrote:

Regardless of the technology, this is what I would do:

Post maintenance notices on the site upfront. This way the customers know what to expect and will not be disappointed if they try accessing the site, while you are updating it, and if they encounter errors.

Communicating and talking to the customers helps.

If you were going to take only 15 seconds, say it will take 5 minutes. When done, remove the update notice and say, "It is back. Thank you for your patience" or something on those lines.

Surya Avantsa

Hi, I agree with Surya,

also you can use more similar page for a graceful degradation on the website if there are some problem connecting to other services (like a credit card payment services) so your customers can create and save the carts and will not be scared by an "OOPS your payment is lost " thing

Another nice-to-have is a minimal static html website to use if the database connection is not working (if the connection fail just redirect to the static pages)

Simone R. wrote: [...]

Another nice-to-have is a minimal static html website to use if the database connection is not working (if the connection fail just redirect to the static pages)

FWIW, Capistrano can do this for you.

Best,

I know of capistrano's web:disable, but not an automatic switch if db connection can't be made. Any link for how to set that up?

Niels

Niels Meersschaert wrote:

I know of capistrano's web:disable, but not an automatic switch if db connection can't be made. Any link for how to set that up?

No, I was thinking of web:disable. Sorry.

Niels

Best,

Hi Dan,

Depends on the needs/functionality of your app, but the two things I usually incorporate in my web apps to address production environment upgrades/maintenance are:

1) the ability to render a real-time system-wide message to all current users of the app; and

2) the ability to toggle on/off read-only mode for the app in real- time.

To do this I'll usually have a couple of before_filters in ./app/ controllers/application.rb: one that looks for a tmp text file that contains the particular sys_wide msg that you want to display to all current users of the app; and another tmp text file that indicates whether the app should be put into read-only mode.

If the app sees that there's a sys_wide msg to be displayed, then it reads the contents of that sys_wide msg and renders it (via a partial called from the app's layouts, displayed prominently at the top of every screen of the app). Similarly, if the app sees that the app should be put into read-only mode, then the app displays another sys_wide msg about the app being in read_only mode, and disallows all requests other than GETs (usually redirecting POSTs, PUTs, ... to the prev refer'd GET req with a warning msg about being in read-only mode).

So, my typical prod env upgrade/maintenance process is: put up a sys_wide msg ahead of time to let current users know about the upcoming/planned update (usually with the time that it's planned to occur and the est time that it should take to complete); then just before the update, I'll put the app in read-only mode (to disallow data mods to the app during the update); then once the update is completed, whether it required an app restart or not, I turn off read- only mode, and put up a sys_wide msg about the update having been completed (which is itself taken down after a brief period of time).

Jeff

Wow, thanks for all the replies I greatly appreciate it. Looks like I got some reading to do, I have found some capistrano scripts that seem pretty promising. And unicorn also seems really cool, github is using them and I just read their article on how they use it and why it is better.

http://github.com/blog/517-unicorn

I will have to play around with a few of them to see which one better suits me. Thanks again!