Explaining Thruster, a new 37signals gem that speeds up your app

rubyonrails-talk
,
1

Last month, 37Signals introduced Thruster, a “zero-config” gem that makes your web pages load faster by solving various problems that would otherwise require changes in multiple places in your infrastructure.

This post explains what those problems are, how Thruster solves them, and why you might want to use it even if you have a CDN like Cloudflare in front of your app, which already implements all the features that Thruster offers.

1. Puma is a great application server, but a poor web server

Both web and application servers work by receiving requests and responding with content. The difference between them lies in the type of content they specialize in:

  • Web: They are designed to handle static content. That is, content that is saved to a file, like the assets of your app (CSS, JS, images, etc.). The most well-known are Nginx and Apache.
  • Application: They are designed to handle dynamic content. That is, content that your app generates on the fly for that specific request, like HTML or JSON. For a Rails app, these include Puma, Unicorn, and Passenger.

So, when a user types the URL of your app (e.g., https://vinklo.com.br), Puma does a great job at generating and returning the HTML, but when the browser starts requesting the assets it needs to actually render the page… not so much.

1.1 What’s wrong with Puma?

The problem is that Puma does not support two critical features of web servers:

HTTP/2

Without this, browsers are forced to request one asset at a time per connection. And since they are limited to 6-8 connections per domain and most pages have dozens of assets, this means they need to make multiple round trips to fetch everything they need.

Zero Copy

When transferring an asset, Puma has to copy its content from the disk to its memory, and then copy it again from its memory to the network. While this is happening, one of Puma’s threads is “stuck” handling the copy, using CPU time and memory.

A web server like Nginx handles the transfer by leveraging a Linux kernel command called sendfile(). When it receives the request, instead of handling the copy itself, Nginx tells the OS to handle it, which allows it to immediately handle the next request. The OS, in turn, uses the DMA controller to set up a transfer directly between the disk and the network.

This not only saves CPU time and memory but is also much faster. Some files get transferred in less than one-third of the time.

2. The three ways production apps solve this problem

Now that we know why forcing Puma to serve files is a bad idea, the question is, how do we solve this problem?

2.1 Nginx

If you have control of your infrastructure, one option is to use Nginx. How this will be done depends on if you are using Nginx as a load balancer or companion to Puma, and if you need to handle Active Storage files or not.

2.1.1 Running one Nginx process in each VM that has a Puma process

Advantages

  • You can use your cloud host’s load balancer.
  • One less single point of failure in your infrastructure.
  • No need to worry about downtime when the load balancer VM needs to restart to apply security updates.
  • Nginx can serve files straight from the public folder of your app, without creating a copy for itself.

Disadvantages

  • If you are using Active Storage, Nginx wil have to create a local copy of the file. This means that each nginx process will have to create it’s own copy, so you will still have multiple requests for the same asset reaching Puma.
  • If your infrastructure is setup in a way that deploying means replacing a docker container, your Nginx cache will be discarded too, unless you mount a folder from the host machine onto the VM folder that Nginx uses for storage.

2.1.2 Running a single Nginx process, in a dedicated VM, as the load balancer

Advantages

  • Having all assets in a single VM makes it less expensive if you want to pay for attached nVME SSD disk;
  • Active Storage files will be cached in a single request and never reach any Puma instance again;
  • No need to worry about deploys clearing nginx cache if you are replacing Docker containers;

Disadvantages

  • You can’t use your cloud host’s load balancer;
  • It’s an extra single point of failure in your infrastructure;
  • Unless you are paying extra for enterprise features of your linux distro, some security updates will require a restart, which will take your app down;

Neutral

  • You can’t serve asset files directly from disk, since they aren’t there, so you will need to configure nginx to create a local copy the first time it sees them (this is what I do). Or change your deploy process to ensure the load balancer VM has the new assets;

2.2 Traditional CDNs (Cloudfront)

Upload your files to a storage service (e.g., S3), and then let the cloud provider CDN (e.g., Cloudfront) handle serving the files instead of your servers.

Downsides: Extra setup work; your deploy process is more complicated because you have to upload new CSS/JS files to the CDN; and your Active Storage will place extra load on your database because you will have to use public URLs.

2.3 Reverse proxy CDNs (Cloudflare)

Advantages

  • Easiest one to configure. Just enable proxy mode
  • No extra component to monitor
  • Automatic compression of assets and polishing of images
  • Works for both assets and active storage

Disadvantages

  • Caching only starts on the third request.
  • Cache is not shared between data centers and there are 300+ of them, which means 600+ requests will reach Puma for every new asset.
  • Just because something is in the cache it does not mean it will stay there since retention and freshness are different things and we can only control freshness.
  • If you use Active Storage and have a lot of images (eg: ecommerce website) Puma will never stop serving assets.

3. The fourth option: Thruster

Even though it’s distributed as a gem, Thruster is almost entirely written in Go. If you add it to your Gemfile and run bundle open thruster in your terminal, you will see this directory tree:

thruster
|-- exe
|   |-- x86_64-darwin
|   |   |-- thrust
|   +-- thrust.rb
+-- lib
     |-- thruster
     |   |-- version.rb
     +-- thruster.rb
     |-- MIT-LICENSE
     |-- README.md

Aside from the thrust.rb file, which is what you must execute to initialize Thruster, the only other thing the gem contains is an executable for the platform you have the gem installed on. This means that all configuration environment variables must be set as UNIX environment variables, not Ruby environment variables (e.g., .rbenv-vars).

This executable is a proxy server that sits between Puma and your load balancer, providing some of the features that are missing in Puma/Rails:

  • HTTP/2 support
  • Zero-copy file transfers by caching assets after Puma served them once
  • Automatic SSL certificate management with Let’s Encrypt
  • Asset compression

Comparing to the other options:

Advantages

  • Zero configuration
  • It’s a gem and replaces the default puma command, so it works even in Heroku

Disavantages

  • It caches entirely in memory and only assets, not Active Storage.
  • Each Thrust process will need it’s own cache, and each restart for deploy will wipe that cache.
  • Binary is 10MB, which will eat some of the 500MB limit you have on Heroku.
  • Almost no documentation (which is why I’m writing this)

4. Should I use Thruster or one of the three other options?

The answer to that will depend on what your infrastructure looks like:

  • Cloudflare only: You do not rely on Active Storage.
  • Cloudflare + Thruster: You rely on Active Storage and are using a PaaS like Heroku to deploy your app, since those usually won’t let you run multiple processes in a single VM.
  • Cloudflare + Nginx: You rely on Active Storage and built your own infrastructure (using Kamal or by hand) and can afford to have an extra VM as your load balancer and know how to configure Nginx for caching asset and active storage routes.
  • Thruster only: You cannot use a reverse proxy CDN and are using a PaaS like Heroku. Just be aware that Heroku’s router only supports HTTP/1.1, when communicating with clients, so even if you add Thruster you won’t get HTTP/2
  • Nginx only: You cannot use a reverse proxy CDN but you built your own infrastructure.
  • Traditional CDN: Personally, I can’t see a reason why I would ever want to use one. They just make your deploy process and your app configuration more complex.

5. Anything else?

Yes, I’m tracking a bug and one possible bug:

  1. Thruster not working in new projects due to the “irb” gem: #16
  2. Thruster caching doesn’t seem to be working: #17

5.1 Changelog

  • 04/12 - Updated Nginx section to account for Xavier’s comment about reading files from disk and placing Nginx on the same VM as Puma.
  • 04/12 - Added “advantages” and “disadvantages” to Thruster.
  • 04/13 - Grammar fixes
20 Likes
2

I’d say it is common to configure NGINX so that it serves static files directly:

root /.../public;

try_files $uri @app;

location ^~ /assets/ {
  expires max;
  add_header Cache-Control "public";
  break;
}
4 Likes
3

That would require Puma and Nginx to be running on the same VM wouldn’t it? I’m thinking of it running as the load balancer in a separate VM. Also, dealing with Active Storage where Puma streams the file from S3, so they are not in the disk for Nginx to serve even if it’s running in the same VM

4

That would require Puma and Nginx to be running on the same VM wouldn’t it?

we are fine with this


task :upload_assets do
  private_ips = fetch(:web_server_private_ips)
  release_dir = release_path.split.last

  on roles(:assets_source) do
    within current_path do
      private_ips.each do |private_ip|
        execute :rsync, "-aL --delete ./public/ --exclude='log' --exclude='uploads' --exclude='system' deploy@#{private_ip}:/srv/www/myapp/#{release_dir}"
      end
    end
  end
end

You will have to add some more tasks to handle rollback and to choose proper dir structure on the standalone nginx, but that’s easy

5

Ah, I see what you mean.

In the setup I have in mind, NGINX is a reverse proxy in each node.

6

Not much of a fan of setting up like this because I use active storage pretty heavily, which means nginx has to create a copy for itself after Puma streams it from storage.

With one nginx per puma, that means each one needs to have their own copy of the file. So you end up in a similar situation of Cloudflare hitting your servers 600 times until the new CSS/JS file is fully cached.

7

Traditional CDNs (Cloudfront) … Downsides: Extra setup work; your deploy process is more complicated because you have to upload new CSS/JS files to the CDN; and your Active Storage will place extra load on your database because you will have to use public URLs.

Cloudfront can also be set up to have your original site serve as the origin. The only setup required is to 1) create a Cloudfront distribution and 2) set the config.asset_host in your application. It is very much a no muss, no fuss setup.

8

Yeah, I still have to update that part. Just need to find the time to actually do this config so I understand how it works.

9

Hi Breno, excellent article. I’m mostly confused about the above two points.

You said that you should only use the Cloudflare only option if you don’t rely on Active Storage. You also said that Thruster only caches static assets, not dynamic ones from Active Storage. If that’s the case, why would the Cloudflare + Thruster option be a good option if you rely on Active Storage? What Active Storage benefits is Cloudflare + Thruster giving that Cloudflare alone does not?

Also, do you think that Cloudflare + Thruster could be a good option if you built your own infrastructure, say with Kamal?

10

Proxy mode active storage services its files in the same way that Puma does: By moving it from the source location (public/assets for puma and storage bucket for Active Storage) to memory, and then from memory to network.

This means that thruster in theory is perfectly capable of handling and caching both (nginx does). That said, as another dev noted on #17, for now thruster only does caching in memory, so it’s cache won’t work very well for Active Storage.

11

This is great! One thing though,

When transferring an asset, Puma has to copy its content from the disk to its memory, and then copy it again from its memory to the network. While this is happening, one of Puma’s threads is “stuck” handling the copy, using CPU time and memory.

I’m not sure this is always true. In some cases, it looks like Puma will use IO.copy_stream. Under the hood that method will use sendfile, or copy_file_range depending on what’s available. Here’s the source in Ruby.

I’m not sure what conditions need to be true for Puma to use copy_stream, but I think 0 copy static file serving is basically table stakes for Ruby webservers these days.

I think the bigger win is that you can probably have way more nginx threads running on one machine than Puma threads. So theoretically you can serve more requests in parallel.

Just for completeness, IO.copy_stream will release the GVL, so Puma can serve non-IO requests in parallel to static assets.

4 Likes
12

Caddy has builtin HTTP/2 and Zero-Copy support. I am not sure why Thruster is needed. Why not use Caddy as reverse proxy for Puma?

2 Likes
13

Oh good. I was both hoping for and dreading someone who actually contributed to ruby to weigh in. I checked a bit of Puma code and looked around for zero copy and all I found was a gem that was no longer used.

Thanks for the correction Aaron!

1 Like
14

No problem! It’s not obvious unless you know already. TBH I feel we should advertise IO.copy_stream or at least update the documentation. The docs don’t mention “zero copy” at all.

2 Likes
15

Thanks for mentioning Caddy. I had never heard of it. Just checked out and it is an amazing project. I’ll try using it in my next project. So far what I read about Truster and Caddy; it seems that Caddy is more mature and solves similar problems. It would be nice to know what Thruster does better than Caddy.

16

Thruster is written in Go and wrapped as gem. You declare the dependency in your Gemfile. So it is tightly integrated with Rails app. If you don’t want to pollute your application codebase with infrastructure related dependencies, keep them separate from your application code. If you use HiveGrid, it is kept separate from your app.

18

I think that i did not found anywhere is that Thruster need Cache-Control to cache assets and that public_file_server is true

config.public_file_server.headers = { ‘Cache-Control’ => ‘public, max-age=31536000’ }

I lost some time until realise it

1 Like
19

You say “Also, dealing with Active Storage where Puma streams the file from S3”, and then say that you would never use a traditional CDN? What? :slight_smile:

The browser would go straight to the CDN, solving the exact problem. The server would never have to do any GET requests to, lets say s3, and only handle uploads and keeping records.

20

I say not to rely only on CDNs because of the retention problem. If you rely heavily on Active Storage you have dozens or hundreds of thousands of images, they will not stay on the CDN cache because they won’t get hit frequently enough, and puma continue getting hit with image requests.

The reason we added nginx to our infrastructure as the load balancer, replacing our cloud host load balancer, was to fix that problem. We operate an ecommerce website and saw that exact scenario of images that should have been on the cache (we on our logs evidence of them being requested until cached a couple days before) not being on the cache.

If you stream images from active storage the CDN is a start, and helps a lot, but it is not enough. This is way on my recommendation sections I said “Cloudflare + nginx” or “Cloudflare + Thruster”

21

Sure, if you specifically mean Cloudflare, but the statement just says “Traditional CDN”. The retention may not be a problem in other CDNs.