Rails habtm grief am I dumb or rails bug? (desperate ; )

Greetings all,

  I'm having a weird problem with a habtm relationship and honestly
I'm beginning to think I may have stumbled upon some weird bug in
rails 3. Surely I'm crazy though. I've been beating my head against
the wall on this for 3 days, have googled everything under the sun I
can think of and still can't come up with an answer.

Ok, the situation:

I'm creating a Rails app to replace both a Java app and a PHP app
(java application and php front-end). This is going to be a phased
operation with the first phase being the Rails application takes over
registration and billing. In order to do this, the Rails application
must create data in the databases for the Java and PHP apps. The
Rails application itself is using Devise for authentication.

In database.yml I have my standard 3 databases defined and also a
connection defined for the Java apps database.
Here are pieces of the model definitions for the external object (I'm
just creating regular rails models to talk to the external databases):

class Pushbroom::UserAccount < ActiveRecord::Base
  require 'digest/md5'
  require 'base64'

  establish_connection :pushbroom
  set_table_name :user_account
  set_primary_key :id

  has_and_belongs_to_many :user_roles, :join_table =>
'pb_prod.users_roles', :class_name =>
'Pushbroom::UserRole', :foreign_key =>
'user_account_id', :association_foreign_key => 'user_role_id'
  belongs_to :user, :dependent => :destroy

attr_accessible :user_roles, :admin_notes, :enabled, :username, :password_hash, :prefStore, :accepted_tos, :do_not_contact

end

class Pushbroom::UserRole < ActiveRecord::Base

  establish_connection :pushbroom
  set_table_name :user_role
  set_primary_key :id

  has_and_belongs_to_many :user_accounts, :join_table =>
'pb_prod.users_roles', :class_name =>
'Pushbroom::UserAccount', :foreign_key =>
'user_role_id', :association_foreign_key => 'user_account_id'

end

And finally my Rails application user object:

class User < ActiveRecord::Base

  before_validation :capture_plaintext_password, :on => :create
  before_save :create_pushbroom_user_data
  after_save :send_welcome_email

  # Include default devise modules. Others available are:

# :token_authenticatable, :encryptable, :confirmable, :lockable, :timeoutable
and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :trackable, :validatable

  belongs_to :pb_user_account, :class_name =>
"Pushbroom::UserAccount", :foreign_key =>
"pb_user_account_id", :dependent => :destroy, :autosave => true

  # Setup accessible (or protected) attributes for your model

attr_accessible :first_name, :last_name, :username, :dob, :email, :password, :password_confirmation, :remember_me

  validates_presence_of :first_name, :last_name, :username, :dob
  validates_date :dob, :on_or_after => lambda
{ 100.years.ago }, :on_or_after_message => "must be on or after
#{100.years.ago.strftime('%m-%d-%Y')}"
  validates_date :dob, :on_or_before => lambda
{ 13.years.ago }, :on_or_before_message => "must be on or before
#{13.years.ago.strftime('%m-%d-%Y')}"

  def capture_plaintext_password
    @plaintext_password = self.password
  end

  def create_pushbroom_user_data
    pb_user = create_pushbroom_user
    add_trial_subscription_to_pb_user(pb_user)
    pb_user_account = create_pushbroom_user_account(pb_user)
    add_subscription_plan_roles_to_pb_user_account(pb_user_account)
    self.pb_user_account = pb_user_account
  end

  def create_pushbroom_user
    pb_user = Pushbroom::User.new
    pb_user.attributes = self.attributes.slice(
      "email",
      "first_name",
      "last_name",
      "dob")

    pb_user
  end

  def add_trial_subscription_to_pb_user(pb_user)
    subscription = Pushbroom::Subscription.new

subscription.populate_from_subscription_plan(Pushbroom::SubscriptionPlan.find_by_name("TRIAL"))
    pb_user.subscriptions << subscription
  end

  def add_subscription_plan_roles_to_pb_user_account(pb_user_account)
    roles_granted =
pb_user_account.user.subscriptions.first.subscription_plan.roles_granted
    pb_user_account.user_roles = roles_granted
  end

  def create_pushbroom_user_account(pb_user)
    pb_user_account = Pushbroom::UserAccount.new
    pb_user_account.enabled = true
    pb_user_account.password_hash =
Pushbroom::UserAccount.create_password_digest(@plaintext_password,
self.username)
    pb_user_account.username = self.username
    pb_user_account.user = pb_user

    pb_user_account
  end

  def send_welcome_email
    AccountMailer.welcome(self).deliver
  end
end

Seems like it should be pretty vanilla. The ONLY weirdness here is
that they aren't in the native rails database and one of the fields is
named funny in the relations table.

So here's a rails console session where I create a rails user, call
the method to create the external objects, then try to save:

ruby-1.9.2-p180 :001 > def user_fred
ruby-1.9.2-p180 :002?> {
ruby-1.9.2-p180 :003 > :first_name => "Fred",
ruby-1.9.2-p180 :004 > :last_name => "Flinstone",
ruby-1.9.2-p180 :005 > :username => "fflint",
ruby-1.9.2-p180 :006 > :dob => "1986-06-01",
ruby-1.9.2-p180 :007 > :email => "fred@mydomain.org",
ruby-1.9.2-p180 :008 > :password => "badpass"
ruby-1.9.2-p180 :009?> }
ruby-1.9.2-p180 :010?> end
=> nil
ruby-1.9.2-p180 :011 > user = User.new(user_fred)
=> #<User id: nil, email: "fred@mydomain.org", encrypted_password:
"$2a$10$IiEOEoSnXIrP7VJAQYckfOVXuzm7Y5ZGo20ayLpSkHhz...",
reset_password_token: nil, remember_created_at: nil, sign_in_count: 0,
current_sign_in_at: nil, last_sign_in_at: nil, current_sign_in_ip:
nil, last_sign_in_ip: nil, created_at: nil, updated_at: nil,
first_name: "Fred", last_name: "Flinstone", username: "fflint", dob:
"1986-06-01", pb_user_account_id: nil>
ruby-1.9.2-p180 :012 > user.create_pushbroom_user_data
=> #<Pushbroom::UserAccount id: nil, created_by: nil, created_at:
nil, updated_by: nil, updated_at: nil, admin_notes: nil, enabled:
true, username: "fflint", password_hash: "blah blah", user_id: nil,
prefStore: nil, accepted_tos: nil, do_not_contact: nil>
ruby-1.9.2-p180 :013 > user.pb_user_account.user_roles
=> [#<Pushbroom::UserRole id: 1, created_by: "script", created_at:
"2008-11-10 12:10:44", updated_by: "script", updated_at: "2008-11-10
12:10:44", admin_notes: "", name: "user", description: "Generic User
Role", conditional: false>]
ruby-1.9.2-p180 :014 > user.save!
NoMethodError: undefined method `relation' for nil:NilClass
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activesupport-3.0.5/lib/active_support/whiny_nil.rb:48:in
`method_missing'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/arel-2.0.9/
lib/arel/insert_manager.rb:22:in `insert'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/arel-2.0.9/
lib/arel/crud.rb:26:in `insert'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/
has_and_belongs_to_many_association.rb:76:in `insert_record'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/association_proxy.rb:
151:in `send'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/autosave_association.rb:306:in
`block in save_collection_association'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/
association_collection.rb:431:in `block in method_missing'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/association_proxy.rb:
216:in `block in method_missing'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/association_proxy.rb:
216:in `each'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/association_proxy.rb:
216:in `method_missing'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/associations/
association_collection.rb:431:in `method_missing'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/autosave_association.rb:297:in
`save_collection_association'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/autosave_association.rb:163:in
`block in add_autosave_association_callbacks'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activesupport-3.0.5/lib/active_support/callbacks.rb:415:in
`_run_create_callbacks'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/callbacks.rb:281:in `create'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/persistence.rb:246:in
`create_or_update'
... 18 levels...
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/callbacks.rb:277:in
`create_or_update'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/persistence.rb:56:in `save!'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/validations.rb:49:in `save!'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/attribute_methods/dirty.rb:30:in
`save!'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/transactions.rb:245:in `block in
save!'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/transactions.rb:292:in `block in
with_transaction_returning_status'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/connection_adapters/abstract/
database_statements.rb:139:in `transaction'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/transactions.rb:207:in
`transaction'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/transactions.rb:290:in
`with_transaction_returning_status'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
activerecord-3.0.5/lib/active_record/transactions.rb:245:in `save!'
  from (irb):14
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
railties-3.0.5/lib/rails/commands/console.rb:44:in `start'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
railties-3.0.5/lib/rails/commands/console.rb:8:in `start'
  from /Users/gander/.rvm/gems/ruby-1.9.2-p180@sms2/gems/
railties-3.0.5/lib/rails/commands.rb:23:in `<top (required)>'
  from script/rails:6:in `require'
  from script/rails:6:in `<main>'ruby-1.9.2-p180 :015 >

If I remove the role assignment, everything is just peachy (finds,
saves, destroys, etc), but the second I try to save roles everything
blows sky-high with this message that, frankly, I don't get. It knows
its got the roles, there is no nil object that I can tell. . .and
basically if I wasn't already bald I'd be pulling my hair out ; )

Any insight into this is EXTREMELY appreciated!

Gerald

If I remove the role assignment, everything is just peachy (finds,
saves, destroys, etc), but the second I try to save roles everything
blows sky-high with this message that, frankly, I don't get. It knows
its got the roles, there is no nil object that I can tell. . .and
basically if I wasn't already bald I'd be pulling my hair out ; )

A cursory examination of the arel code suggests this might happen if
one of the columns you name doesn't in fact exist

Fred

Fred, thanks for the reply. That would seem to be logical and one of
the first things I checked. The table definition for the relationship
table is (pulled straight from the production db):

CREATE TABLE IF NOT EXISTS `users_roles` (
  `user_account_id` bigint(20) NOT NULL,
  `user_role_id` bigint(20) NOT NULL,
  KEY `FKF6CCD9C617041664` (`user_role_id`),
  KEY `FKF6CCD9C63044D5F0` (`user_account_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Any other ideas (or do you see something I don't)? This one is
driving me nuts and has me totally shut down : /

G

Another note along these lines. . . if I load an existing user with
existing roles, they load fine. It just won't let me add new ones.

G

Fred, thanks for the reply. That would seem to be logical and one of
the first things I checked. The table definition for the relationship
table is (pulled straight from the production db):

CREATE TABLE IF NOT EXISTS `users_roles` (
`user_account_id` bigint(20) NOT NULL,
`user_role_id` bigint(20) NOT NULL,
KEY `FKF6CCD9C617041664` (`user_role_id`),
KEY `FKF6CCD9C63044D5F0` (`user_account_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Any other ideas (or do you see something I don't)? This one is
driving me nuts and has me totally shut down : /

I'd double check the sql statemnts rails generates when it is listing
columns to see if they're going to the right database.
Have you tried stepping though the rails insert process in the
debugger to see where it dies?

Fred

Ok, so I've never put a debugger on a rails app before, wasn't real
excited to do so ; )

But. . I did. I'm still poking around, BUT it LOOKS as though it MAY
(like all the emphasis?) not know that the users_role table is in
pb_prod (external db) even though it DOES know that the two
ActiveRecord objects are and there's explicit declaration in my habtm
key definitions. Details:

in arel-2.0.9/lib/arel/table.rb:101
  It's returning false for table_exists? (well, returns nil because of
that)
That's calling table.rb:121 which says:
  @table_exists? ||= tables.key?("pb_prod.users_roles") ||
engine.connection.table_exists?("user_account_id")

Now, to me it looks like the values for those two or backwards (key
implies key, not table name, and reverse for table_exists?) but. .
key? returns false, and table_exists? sends me to:

activerecord-3.0.5/lib/active_record/connection_adapters/abstract/
schema_statements.rb:20

      def table_exists?(table_name)
        tables.include?(table_name.to_s)
      end

And, don't you know it, tables contains the rails database tables not
the external application's tables.

So. . it IS looking at the wrong database.

Ok, now what? I've tried explicitly stating in the key definitions
for the habtm what schema to look at. Any suggestions on what to look
at from here?

Thanks again!

G

Grr, have had two meetings in the middle of all this, forgive my lack
of coherence. Bottom line is it looks like it instantiates the engine
as the rails database and with no way (that I know of) of specifying
which db to look at for the relation table I'm kind of screwed. It
seems to ignore the fact that I'm using :join_table =>
'pb_prod.users_roles" for the relationship definition (pb_prod is the
external database) and isn't picking up which database to use from
either object (connection :pushbroom).

So it looks to me, at the moment, as though I'm stuck. It does bother
me though that it LOADS the relationships fine. Just seems to be a
problem on inserts.

Again, just spouting what I've found and desperately looking for a
solution. Would prefer to find out I'm just being an idiot instead of
having to implement some solution with seriously high code smell.

Thanks!
G

Short of finding and squashing the bug, one not-too-smelly alternative
might be to make a real model for users_roles, like this:

class UsersRoles < ActiveRecord::Base
  establish_connection :pushbroom
  set_primary_key nil

  belongs_to :user_account, :class_name => 'PushBroom::UserAccount'
  belongs_to :user_role, :class_name => 'PushBroom::UserRole'
end

then you could replace the habtm call with a set of
has_many :throughs, which shouldn't have the same problem (since
you've explicitly told Rails the join table is on the :pushbroom
connection).

--Matt Jones

Good timing, just got it fixed (non-smelly)and you're pretty close to
the mark.

The answer? has_many :through - something I'd never taking much of a
look at, but it is actually a pretty nice feature (even in other
circumstances).

Basically this just allows me to create a model class which represents
the relationship. And since I have a model class for it I can
explicitly specify the database to connect to.

For posterity sake, here's the code:

class Pushbroom::UsersRolesRelationship < ActiveRecord::Base

  establish_connection :pushbroom
  set_table_name :users_roles

  belongs_to :user_account
  belongs_to :user_role
end

class Pushbroom::UserAccount < ActiveRecord::Base

  establish_connection :pushbroom
  set_table_name :user_account
  set_primary_key :id

  has_many :users_roles_relationships
  has_many :user_roles, :through
=> :users_roles_relationships, :source => :user_role
end

class Pushbroom::UserRole < ActiveRecord::Base

  establish_connection :pushbroom
  set_table_name :user_role
  set_primary_key :id

  has_many :users_roles_relationships
  has_many :user_accounts, :through
=> :users_roles_relationships, :source => :user_account
end

And is used thusly:

def add_subscription_plan_roles_to_pb_user_account(pb_user_account)
    roles_granted =
pb_user_account.user.subscriptions.first.subscription_plan.roles_granted
    pb_user_account.user_roles = roles_granted
end

Thanks a ton folks for helping me get this train moving again! All my
tests are passing and it seems to be working, but if you see something
wrong, please do still let me know.

Thanks!
Gerald

Oh, also meant to say I don't know that the habtm thing is a bug (not
a feature either : b). It's not the way I would like to see it work
but the rails code that I looked at just flat doesn't make allowances
for m2m relationship tables outside of the rails db that I could tell.

Gerald