Postgres doesn’t support null bytes in string columns. However, null bytes do show up in requests (for whatever reason) and at the moment result in a 500 error, as in the following example:
class BooksController < ActionController::Base
def show
@books = Book.find_by(name: book_params[:name])
render plain: @books.count
end
private
def book_params
params.require(:search).permit(:name)
end
end
class BooksControllerTest < Minitest::Test
include Rack::Test::Methods
def test_index
get "/book", search: { name: "\u0000" }
assert last_response.ok? # ArgumentError: string contains null byte
end
end
Possible solutions
In order to solve the problem in our applications, we discussed multiple strategies on how to handle null bytes in our application. In particular, we tried to figure out where the best place is to handle them:
- Rack Middleware: could come with quite an overhead and
Model.find_by(attribute: "test\00")still raises - case-by-case:
normalizescould be used to sanitise certain attributes.class User < ApplicationRecord normalizes :email, with: ->(str) { str&.delete("\u0000") }) end. However, we would have to remember to apply this change everywhere and a query likeUser.exists?(["email = ?" , "test\00"])will still raise
For our solution, we’ve defined a set of requirements:
RSpec.describe 'Postgres string null byte support' do
let(:search_term) { "test\00" }
context 'when quering directly on the database' do
let(:query) do
ActiveRecord::Base
.connection
.execute("SELECT * FROM materials where materialcode = '#{search_term}'")
end
it 'raises an error as postgres does NOT support null bytes' do
expect { query }.to raise_error(ArgumentError).with_message('string contains null byte')
end
end
context 'when using ActiveRecord' do
context 'with attribute-aware methods' do
let(:query) { Material.where(name: search_term) }
it 'does NOT raise an error' do
expect { query }.not_to raise_error
expect(query.first).to be_nil
end
end
context 'without attribute-aware methods' do
let(:query) { Manufacturer.where('manufacturers.name ILIKE ? '
, search_term) }
it 'does NOT raise an error' do
expect { query }.not_to raise_error
expect(query.first).to be_nil
end
end
end
end
In the end we decided to implement a new OID::PostgresString class:
module ActiveRecord
module ConnectionAdapters
module PostgreSQL
module OID
class PostgresString < Type::String
def serialize(value)
super(PostgreSQL::StringUtils.strip_null_bytes(value))
end
private
def cast_value(value)
super(PostgreSQL::StringUtils.strip_null_bytes(value))
end
end
end
end
end
end
which then gets registered in our PostgreSQLAdapter patch:
# config/initializers/postgres_string_null_byte_support.rb
module ActiveRecord
module ConnectionAdapters
class PostgreSQLAdapter < AbstractAdapter
class << self
alias original_initialize_type_map initialize_type_map
def initialize_type_map(m = type_map)
original_initialize_type_map(m)
register_class_with_limit m,'varchar', OID::PostgresString
end
end
ActiveRecord::Type.register(:string, OID::PostgresString, adapter: :postgresql)
end
end
end
This then allowed us to strip null bytes in all finders and assignments.
Where do you think makes most sense to implement a fix for this? Maybe in the PG adapter as a configurable option (similiar as the ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.decode_dates flag)? Or rather in ruby-pg?
References
Their were also various issues about this already:
- #save raises exception on strings containing NULL byte in postgres · Issue #30730 · rails/rails · GitHub
- ActiveRecord::Base#create raises ArgumentError: string contains null byte · Issue #26891 · rails/rails · GitHub
- Globally strip null bytes for application · Issue #42696 · rails/rails · GitHub
- String contains null byte · Issue #50166 · rails/rails · GitHub
I also made a talk about this problem with more detailed snippets: https://github.com/renuo/postgres-null-bytes-talk/blob/main/presentation.pdf
