Large (10%) increase in memory used after upgrading to Rails 6.0 - Has anyone seen similar?

Hi all!

At my work, we run a relatively large Rails monolith. By count of the rake stats task, we have nearly 1.5 million LOC :slightly_smiling_face:

For the past year and change, we’ve been on a journey to upgrade our app from Rails 4.2 to the latest version, with the goal of running on the tip of main eventually. We recently completed upgrading to 6.0, which was a huge milestone for us!

One thing that we noticed, however, was that after upgrading to 6.0, our app started to use about 10% more memory than it had been using before, when we were running 5.2. Unfortunately, the measurement is not entirely airtight, since other code, features, and bug fixes were also being merged. However, we identified a noticeable jump in memory usage right around the time we merged the upgrade, and the upgrade was the most significant change to be merged at that time, so we feel pretty confident concluding that upgrading to 6.0 is at least a major factor. We’ve also continued to merge code since then and have not noticed any other spikes.

We’ve been trying to track down what is causing the memory increase so that we can hopefully reduce it. We’ve so far used the typical tools (derailed_benchmarks, memory_profiler) to try to analyze where the memory is going, but haven’t been able to find anything conclusive or any smoking gun.

Our app obviously is not a typical Rails application, any way you look at it. But I wanted to ask if anyone reading this has worked on a similarly sized monolith and seen similar increases in memory usage on Rails 6.0? And if you have, what did you do to address them?

Feel free to message me if you’d rather talk privately. My Twitter is @aergonaut, and my Discord is aergonaut#1001.

Thanks :wave:

3 Likes

I don’t remember where I saw it, but it is probably Action MailBox. Try to remove its load from the initial Rails boot and check memory consumption again.

1 Like

Did you also upgraded Ruby version or just Rails?

We did a few upgrades from 5 → 6 and did not notice any large memory increases.

Can you share some more data:

  • graphs of memory usage
  • graphs of ruby heap usage
  • memory profiler output

At Discourse we use GitHub - discourse/prometheus_exporter: A framework for collecting and aggregating prometheus metrics which gives us access to graphs like this: (but other stuff like skylight / newrelic may give you similar info)

Graphing all this data can shed quite a lot of light.

Also recommend: https://samsaffron.com/archive/2019/10/08/debugging-unmanaged-and-hidden-memory-leaks-in-ruby … maybe what you are seeing is a template compilation leak.

2 Likes

@samsaffron Thanks for the reply! I think I can definitely get this data to share. Will post back when I have it.

@megatux We just upgraded the version of Rails. We’re behind on the Ruby version, running 2.7.1. We’ll be upgrading to 2.7.4 soon, but that did not happen around this time.