FasterCVS and Rails

rubyonrails-talk
1

Hello all, I'm trying to import some csv data (that has latin characters) using faster csv but I keep getting a MalformedCSVError: "Unclosed quoted field on line 1." "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in `shift'"

I've tested the same code outside of the Rails application an it works fine. Can anyone help me?

Here is the code:

csv_options = {   :headers => false,   :return_headers => false,   :skip_blanks => false }

csv_file_name = File.dirname(__FILE__) + "/../../Files.TXT" data = "" FasterCSV.foreach(csv_file_name, csv_options) do |row|   data << row.to_s + "<br>" end

And the first two lines in the file: "20.1","Maria José","9" "10.2","José Maria","10"

I'm using fastercsv 1.2.3 and rails 2.1.2

Note: I've tried adding $KCODE = "utf8" but it still does not work.

Thanks. Best regards, Migrate

2

Hi Migrate,

3

Hello Bill,

I'm not sure I understood you. Can you explain further? The file content has only double quotes.

Best regards, Migrate

bill walton wrote:

4

Dharmdip Rathod wrote:

it can be issue with your rails version for current application.

Hello Dharmdip, Do you know any issues between faster csv and rails?

Best regards, Migrate

5

Yes i had it , my rails version was issue . FasterCSV was running smooth on my local but i deployed my application to server FasterCSV was not running so please check server operating system , and install gem related to OS. If your problem is not solved feel free to ask ! ba bye....

6

First off, a quick note on $KCODE - the values it takes are a little weird. 'utf8' is not valid; you're looking for one of these:

None (n or N) EUC (e or E) Shift_JIS (s or S) UTF-8 (u or U)

(taken from Gray Soft / Not Found )

You can also pass an :encoding option (same choice of values) to FasterCSV. I'd also make sure that the file is really UTF8, as many CSVs produced by Excel will instead be in ISO8859-1 (technically, an incompatible MS variant of that standard [CP1252]). You may have to crunch the data somewhat to get it to behave, as the MS Office products are also notorious for stuffing "smart quotes" and other non-standard characters into files...

--Matt Jones

7

Hello all,

I've update to Rails 2.3.2 and the problem remains. I've also created a new application, and added just one controller and FasterCSV to environment.rb (no other plugins are present in the application) but still it does not work.

I already said here that the error only happens in rails. I also found out the following weird behavior: this error only happens if the latin character is in the last two positions in the string. Example 1 (Not working): "1","Maria José","9" Example 2 (Not working): "1","Helder Serrão","9" Example 3 (working): "2","José Maria","10"

Also I checked that it works if the latin caracters exist only in the last column: Example 4 (working): "1","9","Maria José"

Note: I've removed the KCODE part from the code.

8

This confirms my original suspicion; your file is almost certainly in either ISO8859-1 or Windows-1252. The accented characters are being misread, as Ruby's default value (or maybe Rails?) is $KCODE = 'u'. So the accented letters are interpreted as the first character of a 3- byte UTF-8 sequence.

You might want to try passing the :encoding => 'n' option to FasterCSV and see what that does; getting Windows-1252 characters to display correctly on other platforms is left as an exercise.. :slight_smile:

--Matt Jones

9

If it is indeed the case that your accented characters are coming out in ISO8859-1, consider converting the character set in your input first:

csv_data = Iconv.iconv('utf-8', 'ISO_8859-1', file.read).to_s,

Frankly the whole thing seems rather like black magic to me, but the above seems to solve the problem I was having with Excel files in a CSV importer that I've recently created.

-Matt

10

Hello all, I'm trying to import some csv data (that has latin characters) using faster csv but I keep getting a MalformedCSVError: "Unclosed quoted field on line 1." "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in `shift'"

[snip]

I'm using fastercsv 1.2.3 and rails 2.1.2

You may want to try fastercsv 1.5. It was released just a few days ago.

I updated our project to use trunk back in May because of a similar problem. Jonathan

11

Hello Jonathan,

Thanks for the info but I cannot upgrade to 1.5 because I'm using ruport and ruport only accepts 1.2.3.

Best regards, Migrate