Converting data to utf8 to load into an ActiveRecord?

rubyonrails-talk
1

Hi,

I’ve got a text file of data that I need to parse and loaded into my Rails app. I’ve created a migration script the loads and parses the file, but some of the data contains non-utf8 strings, causing an exception.

I’ve found two Windows only ways to convert the strings to utf8, but I would like to deploy my app on heroku. Any suggestions for a non OS specific way to convert?

This is my current implementation using chilkat:

require ‘db/chilkat’

class AddDataSources < ActiveRecord::Migration

def self.encode(str)

strObj = Chilkat::CkString.new()

strObj.appendAnsi(str)

strObj.urlDecode(“utf-8”)

end

def self.up

DataSource.delete_all

data = IO.readlines(‘db/data/data.txt’)

data.each { |line|

cells = line.split(‘,’)

DataSource.create(:code => cells[0],

:authors => encode(cells[1]))

}

end

def self.down

DataSource.delete_all

end

end

2

Iconv?

    irb(main):001:0> require "iconv"     => true     irb(main):002:0> Iconv.conv("utf-8", "windows-1252", "\200")     => "€"

Stefan

3

Thanks, that works.