Migrate column from text to binary (PostgreSQL)

Hello,

So I've realized one of the columns in a table has to be binary
instead of text. However, a straightforward

change_column :my_table, :my_column, :binary

fails under PostgreSQL (9.x series) The error message being:

PGError: ERROR: column "my_column" cannot be cast to type bytea
: ALTER TABLE "my_table" ALTER COLUMN "my_column" TYPE bytea

Now I've being told this problem can be solved with USING clause of
the ALTER TABLE/COLUMN statement, like this:

ALTER TABLE "my_table" ALTER COLUMN "my_column" TYPE bytea USING
"my_column"::bytea;

Given that this is a PostgreSQL extension, I guess this has to go to
the connection adapter and not my code (if I put it there, it won't
work with other DB drivers obviously.)

Any pointers?