Horizontal sharding, activestorage and number of connection pools

rubyonrails-core
,
1

Hello,

I’m trying to setup our rails app (API only) to support horizontal DB sharding: we have a Global db (and corresponding namespace) with some “meta” information for the customer and then each customer has its own Private db (and corresponding namespace). On the Private db we are also able to store files via active_storage without any issues (we don’t need shard support on ActiveStorage::*::Controller - which AFAIK is not there yet).

The issue is on the number of connection_pools rails creates, 3:

  • One connection pool for Global db, owner is ActiveRecord::Base
  • One connection pool on Private db, owner is Private::ApplicationRecord
  • One connection pool on Private db, owner is ActiveStorage::Record

rails console confirms this:

minimize-connections(dev)> ActiveRecord::Base.connection_handler.connection_pools
=> 
[#<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="development" role=:writing>,
 #<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="development" name="private" role=:writing shard=:private>,
 #<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="development" name="private" role=:writing shard=:private>]
minimize-connections(dev)> ActiveRecord::Base.connection_handler.connection_pool_names
=> ["ActiveRecord::Base", "Private::ApplicationRecord", "ActiveStorage::Record"]

What I would like to achieve is to be able to share the connection_pool between Private::ApplicationRecord and ActiveStorage::Record as they are both pointing to the same db. The specific issue is that as we have +100 Private dbs and as we’re running in k8s, every pod creates 2 connection pools for Private db doubling the connections while everything could just work with one connection pool. My first though was to make ActiveStorage::Record inherits from Private::ApplicationRecord instead of ActiveRecord::Base but AFAIK it’s not possible to overwrite that relationship as it is declared in rails/activestorage/app/models/active_storage/record.rb at main · rails/rails · GitHub .

Some example code to better understand the context:

# database.yml
development:
  primary:
    <<: *default
    database: storage/development.sqlite3
  private:
    <<: *default
    database: storage/development_private.sqlite3
    migrations_paths: db/private_migrate

# app/models/global/customer.rb - using "Global" :primary db
module Global
  class Customer < ::ApplicationRecord
  end
end

# app/models/private/application_record.rb - using "Private" :private db
module Private
  class ApplicationRecord < ::ApplicationRecord
    self.abstract_class = true

    connects_to shards: {
      private: { writing: :private }
    }
  end
end

# app/models/private/user.rb - using "Private" :private db
module Private
  class User < Private::ApplicationRecord
    has_one_attached :avatar
  end
end

# config/initializers/active_storage_connection.rb - using "Private" :private db
ActiveSupport.on_load(:active_storage_record) do
  connects_to shards: {
    private: { writing: :private }
  }
end

I’ve tried to move connection pool’s owner to ActiveRecord::Base with the code below, switching shard with ActiveRecord::Base.connected_to(role: :writing, shard: shard) but it’s not working.

# config/initializers/active_record_connection.rb
ActiveSupport.on_load(:active_record) do
  connects_to shards: {
    primary: { writing: :primary },
    private: { writing: :private }
  }
end

Do you have any suggestions for me? I was thinking to propose to modify ActiveRecord::Base - metaprogramming - and make its base class configurable but in case that’s an available option - to contribute upstream - I would need some example/guidance. :smiley:

Thanks, Andrea

2

Here’s a thing that’s worked for me in a similar situation: override the .connection_pool method to delegate to the abstract base class of the “private db”. Something like this should work?

module ConnectionPoolBorrower
  extend ActiveSupport::Concern  

  class_methods do
    def connection_pool
      Private::ApplicationRecord.connection_pool
    end  
  end
end

# in your initializer
ActiveSupport.on_load(:active_storage_record) do
  include ConnectionPoolBorrower
end

And you can do this with any of the Rails records that inherit from ActiveRecord::Base (Action Text, Action Mailbox, …).

You should be able to write a test to convince yourself this works:

assert_same(
  Private::ApplicationRecord.connection_pool,
  ActiveStorage::Record.connection_pool
)
1 Like
3

Hi, first off all thanks for the answer.

Yes it works :tada: - at least the testsuite is green - but didn’t test it in production yet.

Btw to reduce the amount of monkey-patches we have I’ve opened a PR on rails for adding what I was looking for - make ActiveStorage::Record’s superclass configurable - let’s see how it goes :smile:

Thanks again!