How does your team manage a reproducible development environment?

rubyonrails-talk
1

Hi there! Our Ruby on Rails projects are managed by a group of devs, and the team will grow in the near future. As such, we’re looking into ways to make the onboarding experience easier, and reduce the amount of “works on my machine” problems as much as possible.

Therefore, we’re wondering what other people/organizations/companies are using for their Rails projects in this regard. Our current history w.r.t. this is as follows:

1. ad-hoc: No versioning at all. Just use what is on your machine.

Has all the problems you expect with bugs and developer confusion caused by incompatible versions between tools.

2. Use rvm/rbenv to manage Ruby, nvm to manage NodeJS

(at the time of Rails 6 with Webpacker/sprockets, and even now that Rails 7 is a thing we need to rely on some NPM deps).

While this works, it requires people to manage these tools separately. They each come with their own way of installing and managing versions.

3. Use asdf to mange Ruby, NodeJS and Postgres.

This way there was one unified way to manage the versions of our tools. This improved the dev experience quite a bit. However, there still are many gems or NPM deps that require some native tool to exist (like ImageMagick, Selenium, libXML, OpenSSH, etc.). Onboarding and managing deps still is not as simple as “run the installer once” because issues will surely crop up and what exactly they are changes over time.

(Aside: we switched from SQLite to Postgres also in dev to reduce problems we had with incompatibilities between the two DBs that were hurting us when moving code between dev and production (such as handling of Decimal rounding, foreign key constraints and transaction guarantees.)

4. Use a Dockerfile to manage a unified “dev machine”

This made sure that we are all using 100% the same versions of all build tooling etc. However, there still are two glaring problems with this setup:

  • Sychronization of source code between the host computer and the running docker image is slow, especially on mac. This is very annoying and slows down the development process.
  • Maintaining a docker-based setup is quite complex. Currently, the docker image has a very low bus-factor in our organization (essentially I’m the only person that truly understands what is going on). It has also happened multiple times that the base image changes unexpectedly or (now that we have locked its versions down as much as is possible) needs to change; updating this thing is quite a challenge.

5. Using Nix and Bundix

We’re currently looking into using some build tools that specialize in reproducible builds. Nix is probably the most famous one (and the one we tried out), although there are others.

The nice thing about this setup, is that Nix (specifically, the sub-tool bundix) is able to read the gems from your Gemfile.lock and creates its own dependency DAG that includes all other non-Ruby build tools you might need. Nix guarantees that versions (and therefore compatibility of your thing) is always stable.

While nice in theory, current support for Nix – at least for Ruby apps – is still severely lacking:

  • Installation takes a lot longer than simply running bundler and yarn directly.
  • Gems with platform-specific binaries or versions are not supported. You need to build all the stuff from source.
  • It will install all gems, including the ones only needed in other environments.
  • While theoretically Nix can build docker images, it is impossible in practice: It builds everything from source and pull in all of the build tooling needed for that, and not only for the production gems but also for all dev and test gems. All this stuff ends up in the final project or docker image.

So while we’ve tried quite a number of things now, we’re still looking for alternative solutions. What is your team doing in this regard?

Sincerely,

~Marten / Qqwy

5 Likes
2

We just use RVM with gemsets / bundler and it works fine for us. Not having much experience with it, I assume webpacker uses package.json to manage JavaScript versions, similar to what bundler does with Gemfile.lock.

3

I think the simplest approach these days is to run your backing services(MySQL, Elasticsearch, PG) with docker compose and then your host machine runs ruby and the web server or bin/dev. Controlling ruby via your favorite ruby version manager and then bundler. This keeps all editing on your host machine and then you can easily use your favorite editor as you normally would.

What resources and services do you need in your local environment? How complicated are we talking?

You could also look into CodeSpaces and define the base environment and let your team expand from there. Your team can work in the web IDE or locally in VS code and sync their changes up to CodeSpaces.

1 Like
4

And GitHub - github/strap: 👢 Bootstrap your macOS development system. might be useful if you want to have a tool for it, I haven’t used this but it’s what Github uses and is a successor to one of their old tools(Boxen).

1 Like
5

For quite a while we’ve been using 2+4, but recently switched to 2+6 (EC2 snapshot that is used to spin up new instances on aws and CI, and is converted to-and-from VMware image — portable like Docker but waaaay simpler).

For a normal day-to-day development, rbenv+bundler pretty much does the trick. There’s no need to “manage these tools separately”, since everyone is required to use the same tool and install the same version of ruby. No rvm, no asdf, no chruby - smaller the zoo - fewer the problems. Nothing to manage, too - it’s just a few commands to set up a new machine and you’re done - we even have a provisioning script for that: run it on a clean mac/linux and go grab a coffee, it will install all the necessary stuff like xcode, brew, DBs, redis, ES, and whatnot.

Second layer, that is a Single Source Of Truth, or, as we call it, Reality Check, is the virtual machine that boots the same image as is used on production. It used to be Docker, but in the end K.I.S.S. won - everything we were doing with Docker, we could just as easily do without, so, why overcomplicating things? The image is periodically updated with new versions of used system packages or gems to speed up the roll-out, but it’s pretty much automated as well.

Anyway, you can do development on your machine but your tests must pass on CI, and CI uses the same image as production. Basically, your code is required to not only work on your machine, but also work on CI, on Staging (Testing), and on Production.

Boot up the VMware (or Docker) only when: 1) before the PR, to perform a full test run in a production-like environment, and 2) when everything “works on my machine” but fails on CI or in VM - then we’re into some debugging. Debugging should end with a summary for the rest of the dev team, describing what went wrong, what was the reason, how it was fixed, and here’s the link to KB/wiki in case someone else runs into the same issue.

In a rare case when your local environment runs into unsolvable issue/mismatch between your machine and VM (Docker) — you either:

  • a. Reinstall your whole system and set it up from scratch using the provisioning script, and them move on from there, or:
  • b. Do all development in VM (Docker), until it frustrates you enough to finally give up and go to option “a”.

PS. One of the benefits of using an image vs using Docker in your case would be that bus factor is increased dramatically. After all, it’s just a hard drive copy of an operating system. Run it, do the changes you need, stop, send back to AWS, boot up new instance. Voila! :wink:

2 Likes
6

I am now enjoying a hybrid Docker setup. Installing Ruby is left to the user as we all have our preference.

1 Like
7

I was thinking that you could setup a virtual machine environment template with Ubuntu installed, and everything else needed to code for rails.

Then you can share this vm template to get started quicker. ( and update the template).

I think its a pretty good option.

2 Likes
8

I’m at 1.5, no rbenv and nvm, but all using brew version of ruby, node, postgresql, redis and all tools installed from brew. this approach really saving the MacBook battery for digital nomad, also will ensure we are all using the same tool version at the same time.

Sometime I have to pin some version of brew tools, but it won’t last long.

1 Like
9

Thank you all! Those are some great suggestions!

Whom of you is running a single Rails app, and whom is maintaining multiple smaller services?

Wanting to split up our current monolith (into roughly three clearly separate parts) to allow faster iteration and independent scaling is one of our current plans, and (besides the team itself growing in the near future) another driving force for standardizing the way to manage tooling further, because e.g. these separate parts will probably be upgraded to newer versions of Rails, Ruby, NodeJS and other deps at separate points in time.

10

4 separate platforms and 2 (used to be 9) services, all of which we are currently rewriting into a single Majestic Monolith. :wink: The 7-year-long experiment with separate platforms, services and microservices turned out to be one of those learning experiences that take you on a journey through all 5 stages and leave you a little wiser but with a bitter aftertaste of disappointment.

(Micro)services is not a silver bullet. They have their purpose, but if you’re looking at them in hopes that it will suddenly “fix things” or “make things better”, then don’t. Just don’t. It won’t magically fix your mess. If anything, it will only make it worse. It will expose and surface all the wounds, clutches and bad design decisions (especially DB-related) and will multiply them by the number of projects, and then some.

The problem of logic/structure/code/db sharing and synchronisation of it is the most hindering of many.

Use services only if:

  • they will not share anything in common with other parts of the system and 200% guaranteed to not be impacted by code and/or DB changes in other parts of the ecosystem.
  • (if you’re on a big scale and running cost is a serious concern) you require extensive parallelisation (20+) of processes and the service will be consuming at most 1/10 of the memory compared to the monolith, per instance. Candidates are: data importers/processors, if you’re working with millions of records; parallelised calculations for a long-running tasks (>10s if consumer-facing, >1m if internal); querying multiple different external APIs in parallel to combine their result.
  • services can be/are built in a different programming language(s) and communicate with the rest of the system via API.
  • (if money is an issue) doing so will cut your bill by more than 1/3 (or at least a few thousands $).

Reasons to not go down that road:

  • Common DB. Which means common models. Which mean you’ll have to somehow share and synchronise the models and related code. And tests. And the code that works with those models. No, having a gem for models is not a good idea. Engine too. And don’t even look at git submodule. All of those are very limiting and in the long run it’s just not worth it.
  • js/css/views/other code. Because no matter how hard you try, there will always be changes that impact other parts of the system. Which means you’ll have to propagate them. And make sure it doesn’t break things. Everywhere. Tests are great, but they won’t save you either. Even 100% test coverage and 10:1 test-to-code ratio will still let you down. More than once.
  • Services are expected to depend on each other. Entanglement creeps. And you won’t even notice the moment when your fancy “distributed” “uncoupled” “independent” architecture will bring down all the ecosystem just because docker could not install a deprecated package when deploying a forum.
  • And don’t even get me started on deployment synchronisation for cases when changes on different platforms depend on each other…

TLDR: Been there. Seen that. Do not want. Do not recommend. Sounds good, doesn’t work (c)

2 Likes
11

I’d love to know more about the warning signs you saw in those 7 years, @NeilDouglas . Any blogs or other info? Would love to chat about this.

12

Here’s another tool that just bubbled up in my feed that I figured I’d mention here since it uses nix on the backend GitHub - jetpack-io/devbox: Instant, easy, predictable shells and containers.

13

As much as I prefer to run everything on my local machine with for example bin/dev and overmind; I find docker-compose to be the most manageable.

I normally put a makefile in front of docker-compose to reduce the number of tasks a new joiner needs to perform to get up and running.

Now they just Type make dev first time they start which will copy .env files, install gems into a docker volume and create plus migrate the database.

The idea is for new developers to be able to hit the ground running and be contributing from their first day.

I’ve done this for every company I’ve worked for since I got into docker almost ten years ago.

My personal projects and side projects don’t use docker but for more than one developer it eats time with upgrades and such otherwise.

Just take ruby compilation for example. It takes like ten minutes on my machine. With hundred developers that’s 100*10 minutes and everyone have to do it or they can’t continue working.

A docker pull (I tend to wrap ruby in a development base container specific to the company); takes a few seconds.

That is 10 minutes minus a few seconds saved per developer.

I love working directly on my host as many tools (rubocop) isn’t trivial to get working on the host when the gems are in docker.

I just can’t motivate a business why each developer should manage their own setup.

That’s me, I’m a business before pleasure type of guy.

Also, it isn’t always trivial to keep specific database server versions etc installed.

My best bet if you want to avoid docker is to use asdf which has support (via plugins) for Postgres and redis and more. Still, it isn’t trivial to ensure the asdf Postgres is running before running the tests or rails server.

14

Sounds a bit like Devcontainers. Will have to check it out.

15

I suppose you could distribute an Oracle Virtual Appliance with all dependencies included - we did this at one of the (non-rails) shops I was employed at. I suspect that the world. ha smoved to Docker now though. – H

16

I have been using Docker Compose and a devcontainer for the app. The only things running on my computer are Docker, git, and my editor. It has been extremely stable, easy to manage, and easy to upgrade.

Here’s the thing, especially if you use a Mac: you do not own your computer - the OS vendor does. They decide what goes on it and what works, not you. In the case of Apple, the sanctioned dev environment is XCode to build iOS and Mac apps.

Adopting this mindset helped me very easily give up on the stockholm syndrome I felt with rvm and asdf. They are great hacks, but I do not miss the “90% of the time they work every time” behavior they exhibited (the blame for which I will put mostly on Apple)