find encoding (charset) of an uploaded file

rubyonrails-talk
1

Hi, I'm having users upload CSV files, but find a problem with files having different encodings. My default is UTF-8, so when users upload a ISO-8859-1 encoded file, some characters get munged.

I have a standard file upload field in my view (the form is for an object "import"):

<%= f.file_field :file %>

And in my controller I pass the file param to the model CsvFile:

@csv_file = CsvFile.new(params[:import][:file]) @csv_file.import

In my model, I first write the file to disk and then read it using FasterCSV:

def initialize(file)    @file = file end

def import    write_file    FasterCSV.foreach(tmp_file) do |row|       ....    end end

def write_file    self.tmp_file = "#{RAILS_ROOT}/tmp/csv_files/" + rand(9999999999).to_s

   if file       File.open(tmp_file, "w") do |f|         f.write(file.read)       end    end end

I would like to convert it to UTF-8 before saving, changing the write_file like this:

# was: f.write(file.read) # now becomes:    f.write(Iconv.iconv("UTF-8", charset, file.read))

where charset is the charset of the uploaded file. But I need to find out what it is for the conversion to work. It works when I use explicit "ISO-8859-1", but then only if the uploaded file is actually an ISO-8859-1 file. I need to make it variable, because in many cases, the file will just be UTF-8. Anyone ideas?

2

Hi, I'm having users upload CSV files, but find a problem with files having different encodings. My default is UTF-8, so when users upload a ISO-8859-1 encoded file, some characters get munged.

where charset is the charset of the uploaded file. But I need to find out what it is for the conversion to work. It works when I use explicit "ISO-8859-1", but then only if the uploaded file is actually an ISO-8859-1 file. I need to make it variable, because in many cases, the file will just be UTF-8. Anyone ideas?

Check the content-type header ?

Fred

3

That isn’t going to help to determine the encoding of the content of a file. To be honest, you can only make an educated guess and there are several ways to go about it:

  • The easiest way would be for you to let the user decide. You don’t have to ask them for the encoding per se, you can just ask: what application have you created this file with?

  • Some UTF-8 file have a BOM identifier at the beginning of the file. Mind you, i said “some”, not “all”. However, if the BOM is there, you can assume the file is UTF-8 encoded (http://en.wikipedia.org/wiki/Byte-order_mark))

  • Making your own guess: since most character overlap the different encodings and only some special characters differ, you can scan the strings you’re importing. Dutch for example will only use ë, é, è, … Try and find the most common ones for your language and save them with WindowsLatin encoding. Then import them in your rails app and look how they are seen within rails’ UTF-8 context. You then have the choice of either prescanning the files (if they’re not huge) or doing a streaming import and restart the import if you find out along the way you’re using the wrong encoding.

Best regards

Peter De Berdt

4

hi peter! (and dirk)

Peter De Berdt [2008-08-14 13:39]:

- Making your own guess: since most character overlap the different encodings and only some special characters differ, you can scan the strings you're importing. Dutch for example will only use ë, é, è, … Try and find the most common ones for your language and save them with WindowsLatin encoding. Then import them in your rails app and look how they are seen within rails' UTF-8 context. You then have the choice of either prescanning the files (if they're not huge) or doing a streaming import and restart the import if you find out along the way you're using the wrong encoding.

we're doing a similar thing (including BOM detection) in our automatic encoding guesser. see the documentation [1] or install the 'cmess' gem in case you're interested. unfortunately, the actual heuristics are buried in the source [2] so you'd have to look there for details.

dirk's example might then look like this:

  require 'cmess/guess_encoding'

  File.open(tmp_file, 'w') do |f|     input = file.read     charset = CMess::GuessEncoding::Automatic.guess(input)

    f.write(Iconv.iconv('UTF-8', charset, input))   end

[1] <http://prometheus.rubyforge.org/cmess/classes/CMess/GuessEncoding/Automatic.html&gt; [2] <http://prometheus.khi.uni-koeln.de/svn/scratch/cmess/lib/cmess/guess_encoding.rb&gt;

cheers jens

5

Peter, Jens, Thanks for these great answers. I know already it wouldn't be as simple as checking the content-type, the charset usually not being passed properly in there. Your tactics of looking for common characters should work fine for my purposes, we don't need to be 100% accurate, 99,99% is good enough :slight_smile:

I'll have a look at your gem, Jens, on first looks it seems exactly what I need.

cheers, dirk.