2 ActiveRecord features I can offer: eager loading, optimized updates

rubyonrails-core
#1

Hey all,

So there are a couple features that my team would find useful in
ActiveRecord, that I can volunteer our time to work on. I mainly want
to see if there's going to be support in the Rails core team to these
changes into core before doing a full dev cycle on them.

Please let me know if you support these ideas. If so, my team can work
on implementing them.

1) Eager association loading

Currently, if you have a model like this:

   class Forum < AR
     has_many :posts
   end
   class Post < AR
     belongs_to :forum
   end

Then walk thru these records:

   <% for forum in @forums %>
      Title: <%= forum.post.subject %>
   <% end %>

Currently, this will do an extra SQL select * from post for each loop.

DataMapper (datamapper.org) has an interesting approach: The first
time that a sub-association is loaded, then all associations are
preloaded, since most times you're in a loop (as above). While
the :include option helps with this somewhat, when passing collection
objects around, you often don't know how the associations are going to
be used.

2) Optimized updates

Currently, if one attribute is changed in an AR model, then the entire
model is parsed and sent as an UPDATE statement. This is not really
desirable from a network or database standpoint.

There are already some counters in AR for monitoring changes. Is
there a per-attribute @changed data structure? It seems like this
could just be set to true when updating a column. Then, on save, only
those with the @changed flag are actually converted to SQL.

These are changes we really need for our local copy anyways.
Thoughts?

Thanks,
Nate

#2

My forum example was terrible. Here's what I meant:

   <% for post in @posts %>
      Title: <%= post.forum.title %>
   <% end %>

Not a perfect example, but you get the idea. Ideally, you'd do one
bulk select, similar to how :include works, on the first missing
record

   select * from forums where id in (12, 157, 2 , 34)

You would get the id list from a distinct list of all values in
post.forum_id

-Nate

#3

Currently, this will do an extra SQL select * from post for each loop.

DataMapper (datamapper.org) has an interesting approach: The first
time that a sub-association is loaded, then all associations are
preloaded, since most times you're in a loop (as above). While
the :include option helps with this somewhat, when passing collection
objects around, you often don't know how the associations are going to
be used.

There's certainly value to be found here. I've definitely been in the case where one consumer of a method or an association would quite like a whole bunch of things included, but the other wouldn't.

2) Optimized updates

Currently, if one attribute is changed in an AR model, then the entire
model is parsed and sent as an UPDATE statement. This is not really
desirable from a network or database standpoint.

There are already some counters in AR for monitoring changes. Is
there a per-attribute @changed data structure? It seems like this
could just be set to true when updating a column. Then, on save, only
those with the @changed flag are actually converted to SQL.

I can certainly see where you're coming from and something similar has often crossed my mind. The big issue I have with this one is consistency:
Currently I can look at an object, call save! and I know that what's in the db is what I just saw, whereas if you only changed the attributes you touched then you can't be sure of this. There's cases where the right thing to do is probably just to use locking of one form or another, but I can't help but feel that's a bit overkill for the simple case.

Fred

#4

+1 for this, be sure to read Pratik’s writeup http://m.onkey.org/2007/12/18/partial-updates

It couldn’t replace current behavior right away (Frederick explained why), but it could have its own option (or even a method).

These should, in my opinion, do partial updates by default: update_attribute, update_attributes.

#5

Agreed, especially update_attribute. The name just screams "I'm going to change one attribute" but then it resaves the whole object.

Fred

#6

I don't think we'd want to change this default unconditionally.
Perhaps update_attribute(s) should do a partial update iff the table
had a lock attribute, or the model otherwise declared the desire for
partial update.

#7

Interesting post by Pratik. I don't know how much real-world impact
it would really have, though. I've never run into the situation where
I'm save()'ing a record, followed by doing lots more stuff with it.
Normally you redirect and go to another resource.

Nevertheless, I understand why we couldn't change the default
behavior.

For both of these features, I'd recommend AR::B class settings so the
user can control them:

   ActiveRecord::Base.eager_loading = true
   ActiveRecord::Base.partial_updates = true

That way, it could be set in environment.rb, and overridden on a per-
model basis:

   environment.rb
     ActiveRecord::Base.eager_loading = true
     ActiveRecord::Base.partial_updates = true

   class CouldHaveInconsistency < AR::Base
      partial_updates = false
   end

Finally, if a user wanted to do optimistic locking to better ensure
their partial_updates are ok, then they would do that simply by adding
lock_version as is currently. However, I think the two pieces of
functionality should be kept separate, so the user can choose.

Sound good?

#8

Finally, if a user wanted to do optimistic locking to better ensure
their partial_updates are ok, then they would do that simply by adding
lock_version as is currently. However, I think the two pieces of
functionality should be kept separate, so the user can choose.

Sound good?

This has come up a bunch of times here and doesn't tend to get
resolved. In hibernate this feature is called 'dynamic-update', and
you can have a look at their forums etc to see the potential issues
that people hit. It's worth noting that this feature requires
optimistic locking.

I'm certainly interested in seeing patches which implement partial
updates like this, but it's simply not safe to use unless the object
is locked in some way. Otherwise you have *no* way to know that the
values in a row correspond to a valid object. This isn't something
that rails should allow, the risks of corrupted / invalid data FAR
outweigh the difficulty of adding a locking column.

#9

1) Eager association loading

Currently, if you have a model like this:

   class Forum < AR
     has_many :posts
   end
   class Post < AR
     belongs_to :forum
   end

Then walk thru these records:

   <% for forum in @forums %>
      Title: <%= forum.post.subject %>
   <% end %>

Currently, this will do an extra SQL select * from post for each loop.

That's not completely true true. ActiveRecord has basic query
caching, so (in your example), each forum would only be loaded once.
There's been interest in porting my active_record_context plugin into
ActiveRecord though. I have a writeup on it at
http://activereload.net/2007/5/23/spend-less-time-in-the-database-and-more-time-outdoors.
It's basically a simpler, more manual implementation of DM's
association eager loading. There's been some work done to the eager
including functionality that makes my plugin less useful, but
something more automatic would be welcome.

2) Optimized updates

Currently, if one attribute is changed in an AR model, then the entire
model is parsed and sent as an UPDATE statement. This is not really
desirable from a network or database standpoint.

Jeremy (bitsweat) has a plugin for tracking updates to ActiveRecord
attributes: http://code.bitsweat.net/svn/dirty/README It sounds like
this will be going into core. With that, it'd make it pretty easy to
perform dynamic updates. Koz's concerns are definitely valid
though...

#10

DataMapper (datamapper.org) has an interesting approach: The first
time that a sub-association is loaded, then all associations are
preloaded, since most times you're in a loop (as above). While
the :include option helps with this somewhat, when passing collection
objects around, you often don't know how the associations are going to
be used.

Delayed execution is nice. Definitely worth exploring.

2) Optimized updates

Jeremy Kemper has a dirty tracking plugin that's going into Active
Record shortly. That'll solve the first part of the problem (knowing
what changed). The second part, just saving what changed, is a little
bit more tricky. Records can easily have interdependent fields where a
record is only valid if the two fields together fulfill some
requirement. If you save only changed fields, you risk ending up with
an invalid object. Locking can help mitigate this, but it's something
that needs to be considered as part of the implementation.

#11

> 2) Optimized updates

Jeremy Kemper has a dirty tracking plugin that's going into Active
Record shortly. That'll solve the first part of the problem (knowing
what changed). The second part, just saving what changed, is a little
bit more tricky. Records can easily have interdependent fields where a
record is only valid if the two fields together fulfill some
requirement. If you save only changed fields, you risk ending up with
an invalid object. Locking can help mitigate this, but it's something
that needs to be considered as part of the implementation.

Actually, I'm coming around on this. The common case is that you don't
have validation concerns that implicate multiple attributes. The
common case is that it's nice if someone updating the title of a post
won't overwrite the separate changes made to the body by someone else.
We should certainly make it possible and easy to enforce full-
integrity writes, but I think the work should lie with people doing
that.

Naturally, we can't just change this as existing implementations may
break. But I think we should make partial updates the default for apps
created with 2.1. And then finally the default for all by 3.0. We can
do this by making an option like config.active_record.partial_updates
= true that's only set for apps created post-2.1 and when that option
is not set, we complain in the log about the deprecated status.

This should ensure that old apps continue to work just fine and can
upgrade to the partial-update style at their leisure while everything
new enjoys the revised style.

Worth noting is that the dirty tracking that facilitates this relies
on no changes in place. So post.title << "more" won't trigger a dirty
mark and thus won't include post.title in an update. Which is another
reason to make sure that existing apps don't get the partial updates
before they're ready for it.

#12

Naturally, we can't just change this as existing implementations may
break. But I think we should make partial updates the default for apps
created with 2.1. And then finally the default for all by 3.0. We can
do this by making an option like config.active_record.partial_updates
= true that's only set for apps created post-2.1 and when that option
is not set, we complain in the log about the deprecated status.

I think an ever better and safer option is to add optimistic locking
based on updated_at. Most models have this now, and it's a useful
field (compared to an opaque incrementing integer). That way we can
enable it by default, and not worry about busted validations.

If we really want to we can then add a "I know what I'm doing, do
partial updates without a lock" option, but the fact that we'd be the
first orm to do this makes me question the sanity of even considering
that option :).

#13

Uniqueness of timestamps is not a safe assumption. It will find a way
to bite you, eventually.

#14

> I think an ever better and safer option is to add optimistic locking
> based on updated_at.

Uniqueness of timestamps is not a safe assumption. It will find a way
to bite you, eventually.

Particularly fun when you get the code working smoothly on the dev
box, only to fail randomly on more expensive and faster deployment
server.

Version numbers work better, handled by optimistic locking, and a
cheap way to calculate ETag on your models.

Assaf

#15

This is certainly a risk, but it's much less likely than it seems at
first. The particular values of the timestamp column are just opaque
keys used for checking no one else has updated it. I certainly agree
that a version number is the way to go, but if we wanted to push
partial updates out more widely, this would be a low risk way of doing
it.

#16

In distant, but painful memory, it turned out to be *more* likely than
it seemed at first. Likely enough, in that particular scenario, that
it actually happened once in a couple of weeks. And by virtue of being
unreproducible, this bug caused a lot of hurt.

#17

Given that the entire problem that optimistic locking is trying to solve is multiple people fiddling with something at the same time you'd think the odds of a 'collision' would be relatively likely when there were indeed multiple people touching the row in the database at close together times.

Fred

#18

Nate Wiger wrote:

2) Optimized updates

Currently, if one attribute is changed in an AR model, then the entire
model is parsed and sent as an UPDATE statement. This is not really
desirable from a network or database standpoint.

There are already some counters in AR for monitoring changes. Is
there a per-attribute @changed data structure? It seems like this
could just be set to true when updating a column. Then, on save, only
those with the @changed flag are actually converted to SQL.

It isn't quite the solution to the problem, just a hack around it, but
it solves the problem of saving just the pieces of a model I need. See:

http://lists.rubyonrails.org/pipermail/rails/2006-May/039499.html

But that didn't work for me exactly. I had to include some of the model
associations and some of the columns that require validations, like
"enforces_length_of :title" for instance. Adding that to the :select =>
[] and doing an update_attribute instead of a object.save successfully
saved just want I wanted to without saving the entire model object.

position = params[:position]
if section = @stuff.sections.find(params[:id], :select => 'sections.id,
sections.bogus_id, sections.title, sections.position')
  section.update_attribute(:position, position) if section.position !=
position
end

This is what I ended up with. All of these objects are owned and edited
by individual users of the system so I think its safe to assume sure
there won't be any concurrency issues.

#19

Brian Mcquay wrote:

It isn't quite the solution to the problem, just a hack around it, but
it solves the problem of saving just the pieces of a model I need. See:
...

I posted my previous comment prematurely. Upon further testing it isn't
working properly. Please disregard my previous comments.

#20

>> This is certainly a risk, but it's much less likely than it seems at
>> first.

> In distant, but painful memory, it turned out to be *more* likely than
> it seemed at first. Likely enough, in that particular scenario, that
> it actually happened once in a couple of weeks. And by virtue of being
> unreproducible, this bug caused a lot of hurt.

Given that the entire problem that optimistic locking is trying to
solve is multiple people fiddling with something at the same time
you'd think the odds of a 'collision' would be relatively likely when
there were indeed multiple people touching the row in the database at
close together times.

Fred

I've never heard it put so well. I'm going to steal this if you don't
mind. :wink:

@DHH:

Worth noting is that the dirty tracking that facilitates this relies
on no changes in place. So post.title << "more" won't trigger a dirty
mark and thus won't include post.title in an update. Which is another
reason to make sure that existing apps don't get the partial updates
before they're ready for it.

That's pretty drastic (in a good way :wink: ). Ruby would be a nicer
place (sometimes) with the option of cheap, immutable Strings of
course. I'm in the same boat... there's several options, and none of
them nice. Original-value comparison (memory requirements go up, slows
down having to #dup everything), immutable conventions (not "safe"),
writing your own immutable string (what a rabbit-hole!)?

If you can manage to persuade the community on ignoring the mutable
methods by convention, I'll be happy to follow your lead... :smiley: