Why "Circular dependency detected while autoloading constant" vs. NameError after remove_const?

rubyonrails-core
1

Notice how a class that is not available is a NameError and a constant unloaded from ActiveSupport::Dependencies or via remove_const called on the module is a RuntimeError about a circular dependency…

The class I unloaded has no references to or from anywhere else.

Just to show how a missing constant behaves normally:

Loading development environment (Rails 4.0.2)

2.1.0 :001 > Nothing::Nothing.to_s

NameError: uninitialized constant Nothing

from (irb):1

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:90:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:9:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands.rb:62:in `<top (required)>’

from bin/rails:4:in `require’

from bin/rails:4:in `’

Now, how a missing constant behaves after being undefined by Rails 4.0.2:

2.1.0 :002 > Object.const_defined? ‘MyModule::MyClass’

=> false

2.1.0 :003 > MyModule::MyClass.to_s

=> “MyModule::MyClass”

2.1.0 :004 > Object.const_defined? ‘MyModule::MyClass’

=> true

2.1.0 :005 > ActiveSupport::Dependencies.remove_constant MyModule::MyClass

=> MyModule::MyClass

2.1.0 :006 > Object.const_defined? ‘MyModule::MyClass’

=> false

2.1.0 :007 > MyModule::MyClass.to_s

RuntimeError: Circular dependency detected while autoloading constant MyModule::MyClass

from /path/to/gemset/activesupport-4.0.2/lib/active_support/dependencies.rb:461:in `load_missing_constant’

from /path/to/gemset/activesupport-4.0.2/lib/active_support/dependencies.rb:184:in `const_missing’

from (irb):7

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:90:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:9:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands.rb:62:in `<top (required)>’

from bin/rails:4:in `require’

from bin/rails:4:in `’

2.1.0 :008 > load “#{Rails.root}/app/an_autoloaded_dir/my_module/my_class.rb”

=> true

2.1.0 :009 > MyModule::MyClass.to_s

=> “MyModule::MyClass”

2.1.0 :010 > MyModule.send(:remove_const, :MyClass)

=> MyModule::MyClass

2.1.0 :011 > Object.const_defined? ‘MyModule::MyClass’

=> false

2.1.0 :012 > MyModule::MyClass.to_s

RuntimeError: Circular dependency detected while autoloading constant MyModule::MyClass

from /path/to/gemset/activesupport-4.0.2/lib/active_support/dependencies.rb:461:in `load_missing_constant’

from /path/to/gemset/activesupport-4.0.2/lib/active_support/dependencies.rb:184:in `const_missing’

from (irb):23

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:90:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:9:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands.rb:62:in `<top (required)>’

from bin/rails:4:in `require’

from bin/rails:4:in `’

2.1.0 :013 > load “#{Rails.root}/app/workflow/my_module/my_class.rb”

=> true

2.1.0 :015 > MyModule::MyClass.to_s

NameError: uninitialized constant MyModule

from (irb):26

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:90:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands/console.rb:9:in `start’

from /path/to/gemset/railties-4.0.2/lib/rails/commands.rb:62:in `<top (required)>’

from bin/rails:4:in `require’

from bin/rails:4:in `’

2.1.0 :016 > MyModule::MyClass.to_s

=> “MyModule::MyClass”

I’m curious what the circular dependency would be there, since there are no other references to the class, afaik.

Thanks!

2

Sorry ignore the :013-:016 there. Try to multitask / clean-up that and failed.

3

Ok, just bumping this topic because it is actually affecting us now. (What timing, eh?)

When we deploy new code to passenger-enterprise-server-4.0.37 with Rails 4.0.2, at times things have been remove_const’d and I’m assuming that passenger is expecting that Rails autoloading with autoload the constant on second reference? I would at least.

I’m probably just not remembering something, but I’m curious why Rails is not autoloading on reference after a const is unloaded? It seems that instead of raising “Circular dependency detected while autoloading constant…” it would just reload the file, but maybe it doesn’t know which to reload?

4

Let me give you a quick answer in case it rings a bell.

Since the email mentions passenger I assume the app is experiencing this in production mode (if that's wrong we'll do a second pass :).

In production mode, by default, constant autoloding is present to be able to load the application, but class reloading is disabled. Once the User class is loaded, that's it, it is live for the entire live of the process.

In production mode Rails is not supposed to remove constants.

What is removing constants?

5

In addition, sometimes we get InverseOfAssociationNotFoundError in the same scenario, and the related inverse_of’s are setup correctly (or at least I try to- wrote a test similar to this to check that we have them setup correctly as part of a test similar to this one Model and factory_girl factory validation · GitHub though it needs work) between has_one and belongs_to (inverse_of’s on both) only once after hot deploy and then is fine. Related S.O. question I just posted: http://stackoverflow.com/questions/21938347/inverseofassociationnotfounderror-and-circular-dependency-detected-in-rails-4

6

Xavier,

I got these report from another developer on the team (was looking into it last night just from the model side- have not yet reproduced myself but he has reproduced several times now) and I think he is running in development mode locally according to the paths on the traces I saw. (I doubt he’d be running production locally.)

Thanks for your quick assistance,

Gary

7

OK.

The basic idea of the circular dependency control is to detect this:

$ cat a.rb B A = 1

$ cat b.rb A B = 1

If some code triggers autoloading of A (or B), you'd get an infinite loop. To halt the loop you'd need to get an object defined to the constant somehow, but the code wouldn't ever reach that point.

In any case, if you guys could open an issue in GitHub with a minimal way to reproduce the problem we'd look into it.

8

One more answer to your question for clarification- we are not removing constants in our app ourselves- that is just something I was doing in the console and noticed earlier in the thread. It was a coincidence that I happened to see that, as seems to often be the case with me over the past year or two.

Looking at local passenger-enterprise-server-4.0.37 gem code and grep -r’ing for remove_const I see references in Boost header (.hpp) files but the .rb references aren’t anything related to this, so unless it is something else, I think that maybe it is a Rails 4.0.2 issue when running under passenger-enterprise-server-4.0.37 with lots of models and interrelations via associations, etc.

9

Xavier,

Thanks for the example, but I’m confused because:

  1. Classes couldn’t load in the first place if there was no way to load them because of interdependency problems. Rails initially autoloads these files without problems.

  2. I would think that you shouldn’t get an infinite loop in this case if you keep track of what needs to be loaded, and because of that, it is aware that there is a circular reference I assume by checking that without letting a SystemStackError occur before you find that out.

So why not:

A. Keep that list of things to load.

B. Try to load them in as sane an order as possible based on dependencies that are known.

C. For things with circular dependency, put those in a bucket of unknown order (hopefully there aren’t too many) and use the “try and try again” approach iterating through things that need to be loaded, only stopping when you cannot load anything else in the list to be loaded in that iteration.

I know it is much more complicated than that and there is a lot of history that goes with Rails autoloading and why things are the way they are, but you have to admit, it makes no sense that Rails can autoload everything on start and you can manually load (via load) files after you remove_const’d it, but it just “gives up” if it detects a circular reference when trying to autoload a single (class) constant you remove_const’d.

But my lame suggestion there I know does not help as much since you need something to use to reproduce. I’m just not sure yet how to do that simply.

The more worrying thing to me is that this is not just a “Circular dependency detected while autoloading constant” problem, but also possibly (unless is a different problem that is also related to autoloading) InverseOfAssociationNotFoundError problem, so it is definitely mucking things up.

10

That needs an answer beyond “quick” :). Let me write to you later.

But still, with a way to reproduce if possible we would have something concrete to study.

11

1. Classes couldn't load in the first place if there was no way to load

them because of interdependency problems. Rails initially autoloads these files without problems.

By now the error has been reported in development mode. In development mode Rails does not eager load models.

But in any case, in general, this depends ultimately on the code being loaded in the runtime environment which without more information could have conditional code or anything possible in Ruby. Without a way to reproduce we cannot move forward in understanding what happens in that application.

2. I would think that you shouldn't get an infinite loop in this case if

you keep track of what needs to be loaded, and because of that, it is aware that there is a circular reference I assume by checking that without letting a SystemStackError occur before you find that out.

dependencies.rb keeps track of files loaded to autoload constants indeed, that's what allows Active Support to raise that particular error. That way it can avoid the infinite loop (infine nested calls to const_missing that never return to be specific), fail fast, and give a meaningful message.

A. Keep that list of things to load.

B. Try to load them in as sane an order as possible based on dependencies that are known.

The dependencies in a given file cannot be known a priori. You need to execute and let const_missing be triggered as needed.

C. For things with circular dependency, put those in a bucket of unknown order (hopefully there aren't too many) and use the "try and try again" approach iterating through things that need to be loaded, only stopping when you cannot load anything else in the list to be loaded in that iteration.

That optimistic apprach is used for *unloading* constants. dependencies.rb keeps track of the constants that were autoloaded in a flat collection (autoloaded_constants). Then it iterates over the collection to remove them without a topological sort or anything fancy. If the constant to be removed was already removed fine, move along.

But you cannot do eager loading as an alternative to const_missing.

First, eager loading does not solve the problem I wrote above with a.rb and b.rb, that circularity is fundamental.

Second, even without circularities it would be too costly in development mode. In production mode is considered to be worthwhile mainly for thread-safety and to leverage copy on write.

Third, const_missing is triggered when you eager load files. In production mode dependencies.rb still plays its role by autoloading constants, because when a file is interpreted, even if it is being eager loaded by Rails, constants not yet loaded can be present at file or class level (say).

Finally, you cannot evaluate a file, and if something errs mark it for a second try, because the error condition (or missing constant) could be at any point in said file and that would generally imply executing code twice: the code that runs fine until you reach the problematic spot. Evaluating code twice is not acceptable, you cannot do that.

Anyway, let's see if you guys can track it down and we are able to discern if there's something to improve in dependencies.rb or the application should do something in a different way. The error message includes which is the problematic constant name. That's the key data to start debugging.

When dealing with constant autoloading it is very important to remember that dependencies.rb does NOT emulate Ruby algorithms for constant name resolution (that's just not possible with const_missing). dependencies.rb has rather its own rules inspired by those algorithms.

12

Xavier,

Thanks very much for spending the time to respond fully to this.

I think that this may have started when we started using inverse_of everywhere we could in our models. Even though inverse_of doesn’t have references to the other model constants in the models themselves, perhaps on the AR side (I’ve not looked) there are references to constants that are interfering with constant reloading after models are updated. That is off-the-cuff, and I’m probably wrong.

Not sure when we will be able to successfully provide a simple example to reproduce, but I might spend some time today seeing if I can do it independent of passenger and just using AR with examples of various test models using inverse_of and if I have any luck reproducing, will link to example setup. I bet it will take more time to reproduce, though.

Thanks again for your help!

Gary

13

Awesome Gary, let’s if there’s a bug related to inverse_of. Please count on my for anything I can help.

14

Is using inverse_of “everywhere you can think of” still a good approach in rails 4.0.3? We are sort of doing the same (except polymorphic relations), but just doesn’t feel right.

Thanks,

Dmitri.