FasterCVS and Rails

Hello all,
I'm trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
"Unclosed quoted field on line 1."
"/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift'"

I've tested the same code outside of the Rails application an it works
fine.
Can anyone help me?

Here is the code:

csv_options = {
  :headers => false,
  :return_headers => false,
  :skip_blanks => false
}

csv_file_name = File.dirname(__FILE__) + "/../../Files.TXT"
data = ""
FasterCSV.foreach(csv_file_name, csv_options) do |row|
  data << row.to_s + "<br>"
end

And the first two lines in the file:
"20.1","Maria José","9"
"10.2","José Maria","10"

I'm using fastercsv 1.2.3 and rails 2.1.2

Note: I've tried adding $KCODE = "utf8" but it still does not work.

Thanks.
Best regards,
Migrate

Hi Migrate,

Hello Bill,

I'm not sure I understood you. Can you explain further?
The file content has only double quotes.

Best regards,
Migrate

bill walton wrote:

Dharmdip Rathod wrote:

it can be issue with your rails version for current application.

Hello Dharmdip,
Do you know any issues between faster csv and rails?

Best regards,
Migrate

Yes i had it , my rails version was issue .
FasterCSV was running smooth on my local but i deployed my application
to server FasterCSV was not running so please check server operating
system , and install gem related to OS. If your problem is not solved
feel free to ask !
ba bye....

First off, a quick note on $KCODE - the values it takes are a little
weird. 'utf8' is not valid; you're looking for one of these:

None (n or N)
EUC (e or E)
Shift_JIS (s or S)
UTF-8 (u or U)

(taken from http://blog.grayproductions.net/articles/bytes_and_characters_in_ruby_18
)

You can also pass an :encoding option (same choice of values) to
FasterCSV.
I'd also make sure that the file is really UTF8, as many CSVs produced
by Excel will instead be in ISO8859-1 (technically, an incompatible MS
variant of that standard [CP1252]). You may have to crunch the data
somewhat to get it to behave, as the MS Office products are also
notorious for stuffing "smart quotes" and other non-standard
characters into files...

--Matt Jones

Hello all,

I've update to Rails 2.3.2 and the problem remains.
I've also created a new application, and added just one controller
and FasterCSV to environment.rb (no other plugins are present in the
application) but still it does not work.

I already said here that the error only happens in rails. I also found
out the following weird behavior: this error only happens if the latin
character is in the last two positions in the string.
Example 1 (Not working):
"1","Maria José","9"
Example 2 (Not working):
"1","Helder Serrão","9"
Example 3 (working):
"2","José Maria","10"

Also I checked that it works if the latin caracters exist only in the
last column:
Example 4 (working):
"1","9","Maria José"

Note: I've removed the KCODE part from the code.

This confirms my original suspicion; your file is almost certainly in
either ISO8859-1 or Windows-1252. The accented characters are being
misread, as Ruby's default value (or maybe Rails?) is $KCODE = 'u'. So
the accented letters are interpreted as the first character of a 3-
byte UTF-8 sequence.

You might want to try passing the :encoding => 'n' option to FasterCSV
and see what that does; getting Windows-1252 characters to display
correctly on other platforms is left as an exercise.. :slight_smile:

--Matt Jones

If it is indeed the case that your accented characters are coming out
in ISO8859-1, consider converting the character set in your input
first:

csv_data = Iconv.iconv('utf-8', 'ISO_8859-1', file.read).to_s,

Frankly the whole thing seems rather like black magic to me, but the
above seems to solve the problem I was having with Excel files in a
CSV importer that I've recently created.

-Matt

Hello all,
I'm trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
"Unclosed quoted field on line 1."
"/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift'"

[snip]

I'm using fastercsv 1.2.3 and rails 2.1.2

You may want to try fastercsv 1.5. It was released just a few days
ago.

I updated our project to use trunk back in May because of a similar
problem.
Jonathan

Hello Jonathan,

Thanks for the info but I cannot upgrade to 1.5
because I'm using ruport and ruport only accepts 1.2.3.

Best regards,
Migrate