Request failures: lots of ActiveRecord::Not NullViolation

I created a model with associations. But when I add more than 46 records to the association, the main model can no longer be saved - it fails with

PG::NotNullViolation: ERROR: null value in column "connect_id" violates not-null constraint

I finally figured that with more than 46 records Rails starts to write the transaction in two passes, leaving the foreign_keys empty (Null) in the first step and then later during the transaction an update does fill them in.

But, for this to work we would need deferred triggers, and Rails does not create deferrable triggers, at least not by default. :frowning: (One could certainly fix that one.)

Since the foreign keys in my associations can be resolved (they are no circulars), I was wondering why Rails would do this properly with 46 records, but no longer with 47 records.

And after some search I found it in the code - and changed it:

module ActiveRecord
  module AutosaveAssociation
    module ClassMethods
      private
    
      def define_non_cyclic_method(name, &block)
        return if method_defined?(name)

        define_method(name) do |*args|
          result = true; @_already_called ||= {}
          # Loop prevention for validation of associations
          @_already_called[name] ||= 0
          unless @_already_called[name] >= 3
            begin
              @_already_called[name] += 1
              result = instance_eval(&block)
            ensure
              @_already_called[name] = 0
            end
          end
          
          result
        end
      end
    end
  end
end

By default this works with a boolean, i.e. there is only one cyclic loop allowed. This is clearly not enough. OTOH at some point there are stack overflows in ruby, so it is not clear what exactly is the proper value. For now I have it running with 3, and that seems work for my usecases.

Hmmm, I’m not seeing why this is an issue specifically the 47th time. The current definition of that method doesn’t do anything special to do with the number 46 that I can see. Do you have any ideas why?

Ideally we could boil this down to an executable test case

Well, me neither, and I don’t think it’s always the 47th. :wink:

And I was wondering, just like You, when this hit me the first time, some rails-4 or 5 version. This is a configuration app that creates firewall rules from a network topology, and I was wondering why it started to break only for rather big configurations with lots of flows. Now I rewrote the whole application to support IPv6 traffic, and it hit me again, and it’s still the same problem and the same solution.

It was the 47th record in that case when I discovered the problem (again) and walked thru the revert-list of my app to time-travel-restore the models, and found that after adding the 47th network flow record it could no longer be restored. This may be a different number with different data, or already after a database restore.

But what I guarantee you, is, that it will happen, with a complex-enough layout.

Yes, one could create a reproducible testcase. And if I weren’t busy, I might do that - but for now I must see to it that our elderly people have enough firewood for the winter.

Anyway: have a database in fully normalized form, with some 20 or more related tables, and a bunch of foreign keys (multiple ones between the same two tables - that is the essential point, so the search will hit the same table multiple times [1]) going zig-zag between them (but all resolveable, no circular loops).

Then have one topmost model, and all the others being a tree of associations, all correctly defined and connected/reflected.

Then try to deep_clone that topmost model - and it is only a question of how many records are filled into the tree until it will blow. And I already explained where it blows. (I don’t do testcases where I have logical verification.)

[1] I give you an abbreviated example:

class Flow < HierarchyRecord
  belongs_to :ruleset, inverse_of: :flows
  belongs_to :origin, class_name: 'Region', inverse_of: :origin_flows
  belongs_to :respond, class_name: 'Region', inverse_of: :respond_flows
  ...
end

class Region < HierarchyRecord
  belongs_to :connect, inverse_of: :regions
  belongs_to :forward, optional: true, inverse_of: :regions
  has_many :origin_flows, class_name: 'Flow', foreign_key: :origin_id,
           dependent: :destroy, inverse_of: :origin
  has_many :respond_flows, class_name: 'Flow', foreign_key: :respond_id,
           dependent: :destroy, inverse_of: :respond
  has_many :remoteip_forwards, class_name: 'Forward',
           foreign_key: :remoteip_id, dependent: :destroy,
           inverse_of: :remoteip
  has_many :responding_forwards, class_name: 'Forward',
           foreign_key: :responding_id, dependent: :nullify,
           inverse_of: :responding
  ...
end

class Forward < HierarchyRecord
  belongs_to :ruleset, inverse_of: :forwards
  belongs_to :remoteip, class_name: 'Region', inverse_of: :remoteip_forwards
  belongs_to :responding, class_name: 'Region', optional: true,
             inverse_of: :responding_forwards
  has_many :regions, dependent: :nullify, inverse_of: :forward
  ...
end

Anyway, If you’re really interested, I could send you the intermediate json from the deep-clone. But then, it contains all my network IPs…

Thanks for update and quick reply, It’s work for me, Really appreciate for help. indigocard