How to filter db column(s) from log.

rubyonrails-talk
1

Hi, All!

I have been looking for a way to eliminate the contents of certain large (e.g. pdf files) database columns from my development (and production) logs using RoR 3 with no luck. I found many posts, etc. about eliminating parameters (e.g. password) from the log, but nothing about the contents of the db.

I have two reasons for this, 1) the verbosity and size of the log slow down my development/debugging, and 2) writing these columns definitely slows the operation of the site itself.

Can anyone help?

TIA

Donz

2

I believe you’re looking for config.filter_parameters in application.rb. Note that filter_parameters is an array, so you can add to it with <<

Hope this helps,

Garrett Lancaster

compose-unknown-contact.jpg

3

Don Ziesig wrote in post #973810:

Hi, All!

I have been looking for a way to eliminate the contents of certain large (e.g. pdf files) database columns from my development (and production) logs using RoR 3 with no luck.

Wait, you're storing PDF files in your DB? Stop doing that. Data like that belongs in the filesystem.

Best,

4

Filesystem is write-only.

5

Garrett,

I have config.filter_parameters = [:password, :document_file] in

application.rb. :document_file is the name of the big column. I “puts” the array.inspect and confirmed that both password and document_file are there. The contents of the big column are still showing up in the log.

BTW, I was advised to store these data in the file system instead of

the db, Unfortunately, the file system will be read-only once I deploy and I want to test in the environment that I will be using, so I set the rails tree to read only during some of my development and test operations (which is how I discovered that I needed to use the database for all dynamic data in the first place).

Thanks,

Donz

compose-unknown-contact.jpg

6

OOPS.... Filesystem is READ-only :-[

7

OOPS.... Filesystem is READ-only :-[

So use Paperclip and the S3 plugin, and have at it. You can write anything to S3, probably more files than you will ever have patience for.

Walter

8

Hi All (again)!

I started this thread with a set of requirements and conditions that

I did not state in my original post. Among these are:

  1) The ultimate host will have a read-only file system

which precludes use of paperclip’s default filesystem storage, and;

  2) My client's business/legal reasons preclude the use of

paperclip’s S3 option.

  3) I have added the name(s) of the big columns in

config.filter_parameters but that doesn’t seem to change anything.

Pat Shaughnessy's attachment-in-database-column(s) update to

paperclip (which works great, incidentally) does the job except for the cruft in the log when storing large files.

The cruft tends to make my life difficult when analyzing the log and

seems to impact the speed during the time that the logfile is being written.

If I have to, I'll continue with the current implementation and put

up with the aggravation hoping that production mode will stop most of it (have to figure out where to put the log file, too, due to the read-only fs).

Thanks,

Don Ziesig
9

compose-unknown-contact.jpg donz

          January 10, 2011 6:30 PM

Hi, All!

      I have been looking for a way to eliminate the contents of

certain

      large (e.g. pdf files) database columns from my development

(and

      production) logs using RoR 3 with no luck. I found many posts,

etc.

      about eliminating parameters (e.g. password) from the log, but

nothing

      about the contents of the db.



      I have two reasons for this, 1) the verbosity and size of the

log slow

      down my development/debugging, and 2) writing these columns

definitely

      slows the operation of the site itself.



      Can anyone help?

You want it removed from the “SQL (1.2ms) INSERT…” line in the logs? filter_params will remove it from the “Parameters: {…” line.

I think you’re only option for removing it from the SQL in the log itself is to silence the active record logger whenever you do something that generates those queries.

It’s not an issue in production as production doesn’t log the sql queries…

-philip

10

After a long interlude, I found my problem with config.filter_parameters. I can't count how many hours I have wasted due to a typo :frowning: The original line was:

config.filter_parameters = [:password, :document_file]

the correct line is:

config.filter_parameters = [:password, :documents_file]

This isn't the first time that I have gotten burned by rails' pluralization.

Many thanks for all who commented and pointed me in the right direction.

Don Z.