Active Record - Null Forms for nil primitives and associations

Null Objects:

A while back, Rails 4 introduced the concept of a Null Scope. Taking the form of Person.none (or some_company.people.none), it builds a Null Relation that behaves like a “real” Active Record relation, allowing scope chaining and other interface methods available from a scope (like #count), avoiding need for brittle nil checks. I’d like to discuss extending this a general concept of Null Forms in Active Record.

Let’s start with the obvious issue of mapping SQL NULLs to Ruby nils. There is a conceptual mismatch here. SQL NULL means “unknown value”, while Ruby nil means “no value”. This manifests itself rather painfully in anything from JOIN to NOT IN (…) queries. To make things worse, saving data from a form-based request (the most common case of introducing data into Active Record in most Rails apps) will pass blank strings from an empty form. So now you have a mix of NULLs and blank strings in your database, depending on whether or not a form ever updated the record, even if it was saved without any intentional user input.

Null Form primitives:

It may be possible to design the database to simply not allow any NULL attribute values and instead default the appropriate Null Form:

  • ‘’ for varchars
  • 0 for ints
  • [] for serialized arrays
  • {} for Hstore or other serialized hashes
    Etc.

One could argue, though, that such a design introduces excessive logic and performance penalty on the database layer, and that instead, it should be Ruby which uses the Null Form object whenever the database contains a NULL value. So, for example, if I have a database record in the users table with id=1, name=NULL, and I do User.find(1).name, I’d prefer to get ‘’ instead of nil.

So essentially this is a question of whether Active Record type casting should cast NULL values into their Null Form object of the appropriate type, rather than nil. But I see the obvious issues with this approach as well, notably whether #attributes should return the raw nil or use the casted Null Form, especially for APIs, so perhaps manually using database default values with null: false constraints is still the best way to accomplish this.

Null Form associations:

NULL-to-nil mismatches are yet more painful when dealing with associations, like document.project.account.name. We need to do nil checks everywhere, or use try everywhere, or delegate … allow_nil: true hacks. How much nicer would it be if there was a Null Form association? Imagine the following:

document.project_id => nil
document.project => <#Project id=nil>
document.project.persisted? => false
document.project.null? => true
document.project.account_id => nil
document.project.account.null? => true
document.project.account.name? => ‘’ # (or nil if not also using Null Form primitives)

``

Similar to the Null Primitive concept, the idea is to avoid extraneous nil checks. If the context and data type is known beforehand (and it is in the case of ActiveRecord due to schema introspection, the same logic that allows Active Record to know whether an attribute should be casted to string or integer, for example), having Null Form logic would bring a lot of the benefit of types languages into Rails, while reducing (rather than increasing) the maintenance burden and logical complexity of code. In the case of associations, this information could be inferred from has_many / has_one / belongs_to definitions, and/or introspected from used foreign keys (added in Rails 4.2).

In fact, existing code (notably accepts_nested_attributes) already expects you to manually build a new child instance if it is nil and you want an inline child object form, making you write code like @user.build_profile if @user.profile.nil? .

Discussion:

Both nil primitives and nil associations can be dealt with manually, via either explicit nil checks at point of invocation, or by using the above suggestions (such as database null: false, default: ‘’ for blank string, controller @user.build_profile if @user.profile.nil? for blank associations, etc.) But all of these approach feel tedious and brittle, and (more importantly) add unnecessary complexity to understanding and reasoning about code.

I wonder whether anyone had experience implementing some kind of programmatic Null Form behavior for either attribute primitives, associations, or both? Did you only use application-level solutions similar to above? Or did you try some kind of framework-level approach to dynamically build Null Form objects for handling database NULL values? How well did the rest of your code, as well as Rails handle it? For example, there is no way in Ruby to override the truthiness of a Null Object for if object { a } else { b } type comparisons, so any object other than nil would behave differently, including any custom Null Form.

Or, perhaps, there is already some literature you know, or ideas you have, about the best practices to better handle nil primitives/associations? I’d love reading about it.

Thanks!

Null Objects:

A while back, Rails 4 introduced the concept of a Null Scope. Taking the form of Person.none (or some_company.people.none), it builds a Null Relation that behaves like a “real” Active Record relation, allowing scope chaining and other interface methods available from a scope (like #count), avoiding need for brittle nil checks. I’d like to discuss extending this a general concept of Null Forms in Active Record.

Let’s start with the obvious issue of mapping SQL NULLs to Ruby nils. There is a conceptual mismatch here. SQL NULL means “unknown value”, while Ruby nil means “no value”. This manifests itself rather painfully in anything from JOIN to NOT IN (…) queries. To make things worse, saving data from a form-based request (the most common case of introducing data into Active Record in most Rails apps) will pass blank strings from an empty form. So now you have a mix of NULLs and blank strings in your database, depending on whether or not a form ever updated the record, even if it was saved without any intentional user input.

Null Form primitives:

It may be possible to design the database to simply not allow any NULL attribute values and instead default the appropriate Null Form:

  • ‘’ for varchars
  • 0 for ints
  • [] for serialized arrays
  • {} for Hstore or other serialized hashes
    Etc.

One could argue, though, that such a design introduces excessive logic and performance penalty on the database layer, and that instead, it should be Ruby which uses the Null Form object whenever the database contains a NULL value. So, for example, if I have a database record in the users table with id=1, name=NULL, and I do User.find(1).name, I’d prefer to get ‘’ instead of nil.

A billion times no. If you want the DB to return ‘’ for string columns that didn’t have a value inserted, set the column as “default ‘’ not null”. Doing otherwise means that every client program that interacts with the DB also needs to apply the “NULL means empty string” convention.

So essentially this is a question of whether Active Record type casting should cast NULL values into their Null Form object of the appropriate type, rather than nil. But I see the obvious issues with this approach as well, notably whether #attributes should return the raw nil or use the casted Null Form, especially for APIs, so perhaps manually using database default values with null: false constraints is still the best way to accomplish this.

Null Form associations:

NULL-to-nil mismatches are yet more painful when dealing with associations, like document.project.account.name. We need to do nil checks everywhere, or use try everywhere, or delegate … allow_nil: true hacks. How much nicer would it be if there was a Null Form association? Imagine the following:

document.project_id => nil
document.project => <#Project id=nil>
document.project.persisted? => false
document.project.null? => true
document.project.account_id => nil
document.project.account.null? => true
document.project.account.name? => ‘’ # (or nil if not also using Null Form primitives)

``

Similar to the Null Primitive concept, the idea is to avoid extraneous nil checks. If the context and data type is known beforehand (and it is in the case of ActiveRecord due to schema introspection, the same logic that allows Active Record to know whether an attribute should be casted to string or integer, for example), having Null Form logic would bring a lot of the benefit of types languages into Rails, while reducing (rather than increasing) the maintenance burden and logical complexity of code. In the case of associations, this information could be inferred from has_many / has_one / belongs_to definitions, and/or introspected from used foreign keys (added in Rails 4.2).

In fact, existing code (notably accepts_nested_attributes) already expects you to manually build a new child instance if it is nil and you want an inline child object form, making you write code like @user.build_profile if @user.profile.nil? .

Discussion:

Both nil primitives and nil associations can be dealt with manually, via either explicit nil checks at point of invocation, or by using the above suggestions (such as database null: false, default: ‘’ for blank string, controller @user.build_profile if @user.profile.nil? for blank associations, etc.) But all of these approach feel tedious and brittle, and (more importantly) add unnecessary complexity to understanding and reasoning about code.

I wonder whether anyone had experience implementing some kind of programmatic Null Form behavior for either attribute primitives, associations, or both?

I’ve personally used some of these patterns in specific cases, and they seemed sufficiently straightforward that framework support would have made them LESS clear.

For instance, if you’ve got a string that shouldn’t ever be nil, and you can’t (for some reason) just declare it “default ‘’ not null”, you can do something like this:

class SomeModel < ActiveRecord::Base

def attribute_that_cant_be_nil

super || ‘’

end

end

(similarly with setters, if writing empty strings is not desired)

If you have an association that you want to “pop” into existence when referenced, as in the user profile thing:

class User < ActiveRecord::Base

has_one :profile

def profile

super || build_profile

end

end

With this, you can happily refer to @user.profile and either get the existing record or a newly-instantiated one.

If you were feeling really clever, you could even override a particular accessor (as above) to return a NullAssociationObject, but there’s going to be a lot of plumbing required to make that object work all the places a model normally would…

–Matt Jones