Hello everyone. We are running a proprietary Rails app in my company, currently on Rails 5.2.5, and using the Puma web server configured with 12 threads and a single process in a Dockerized environment. We see the following message logged in our Production environment fairly regularly:
/home/app/web/vendor/bundle/ruby/2.7.0/gems/railties-5.2.5/lib/rails/paths.rb:196: warning: conflicting chdir during another chdir block
This warning seems to sometimes be the precursor to the process having it’s working directory permanently changed, which results in the server threads being unable to access files in their expected location. Container instances that have this happen to them become unresponsive due to these errors and need to be terminated.
In one of the threads of my investigation on this, I discovered this Rails PR which seems to abandon the user of
Dir.chdir with the block syntax, and appears to be in the Rails 6.1 release. So presumably updating to Rails 6.1 and getting this change would fix the issue. I am wondering though - there don’t seem to be any bugs attached to that change, which seems a little odd to me. Wouldn’t you assume that changing the process directory in a potentially multithreaded app, even just for a single line, is highly dangerous when you have no idea what other threads might be running and what they might be doing? Has nobody else been bit by this before, or am I missing something?
One of my other threads traced some of the calls to use of
config_for in some of our initializers, which ends up going through that code path. It seems that using this in an initializer for a multithreaded server causes all of the threads to run through that code at roughly the same time. It seems plausible that once in a while, those threaded calls create a deadlock that results in the process still having the path alteration that was meant to be temporary. For now, I’m testing out getting rid of all of the
config_for calls, and I will look at upgrading the app to Rails 6.1.