N+1 Queries should be the exception, not the easy trap to fall in

When learning rails many years ago, it seemed that understanding and avoiding n+1 queries was both a right of passage and an all-too-common interview question. Fast forward to 2020, and rails still makes it so easy to write, and most learning in a void will still not understand until they hook up a tool like scout, skylight, or new relic what they’re doing.

Something like Dmitry’s solution (https://github.com/DmitryTsepelev/ar_lazy_preload)should be the default, not the easy trap to fall into.

7 Likes

Yes, this also bugs me. Imo something like Goldiloader should be part of Rails core. We’re using it in production since years and it makes Rails feeling much railsier (Rails doing the right thing be default).

4 Likes

Thanks for that resource, I hadn’t tried that gem before. Totally going to give it a spin!

This feature has been proposed at least twice recently: https://github.com/rails/rails/pull/34843 and https://github.com/rails/rails/pull/33527. Both closed as stale though :man_shrugging:t2:

2 Likes

@tenderlove @rafaelfranca thoughts on Goldiloader’s and ar_lazy_preload’s designs? What might be the gotchas of shipping something like that as part of default Rails?

3 Likes

Not sure if this solves what you’re looking for, but I was excited to see it get merged. I’d love it on by default. https://github.com/rails/rails/pull/37400

5 Likes

OMG! I have wanted something like that badly for a while. How exciting!

1 Like

Definitely a potential solution, but it should be the default, with lazy loading being the override option.

Thanks for pointing it out!

There is also the jit_preloader gem that takes a slightly different approach to goldiloader and has some additional features on top of it as well

2 Likes

I haven’t looked at the gem internals closely. Just from reading the README, it seems fine, but I don’t know enough to make an informed judgement :sweat_smile:

1 Like

@nickbender-- use bullet !

it’s still amazing

prefetching at default :-1:

2 Likes

This was a problem I wanted to solve for a long time. In Lucky it is handled well and so far has gotten a lot of positive feedback

In dev and test it raises an error if you forget to preload. In production it allows lazy loading. That way if you accidentally ship a change and don’t have tests for it, it won’t blow up prod.

It also lets you lazy load for cases where it is just 1 query.

Here’s roughly how I’d see it working in Rails:

articles = Articles.all
# Raise error: "The 'user' association was not preloaded. [Link to docs on includes/eager loading]"
articles.map(&:user)

articles = Articles.includes(:user)
articles.map(&:user) # works
author = Author.find(1)
author.avatar # Raises: "The 'avatar' association was not preloaded. [Link to docs on includes/eager loading]"

# But isn't really n+1 so let's have an escape hatch
author.avatar!
# or maybe
author.lazy(:avatar)

I hope this helps! I implemented this in Lucky so happy to talk about implementation or how it works

2 Likes

Thanks, everyone for offering up the many viable solutions!

There are lots of great examples of how to make it easier for devs of all skill levels to not get caught on snags.

I’d argue the best solution is going by the path of least surprise, which means making it clearer what best practices are around loading additional models, and stopping N+1 queries from happening silently. The end goal is to help empower users to make better decisions, with the best case being not having to make decisions at all to have rails be performant in its simplest state.

1 Like

One thing related which we are talking about including in Rails is including rack-mini-profiler by default.

Recently Osama Sayegh finished some internal cleanup that passed @rafaelfranca’s review and this was merged into Rails :confetti_ball:

https://github.com/rails/rails/pull/38701

I personally rely on it quite a lot to find N+1s both in dev and production cause it becomes quite a habit to sneak a look there and notice right away if something is looking odd.

10 Likes