Hello everyone,
I'm scraping a lot of sites for a project, and occasionally the scraped content will have "malformed UTF-8" characters. When the scraped content is processed (basically a database record is created), these characters often don't appear as they're supposed to.
Normally, the following code works great:
str.unpack("U*").collect {|s| (s > 127 ? "&##{s};" : s.chr) }.join("")
But it won't work with these "malformed UTF-8" characters. So I've written the following to handle these characters, but it still isn't perfect. For example, I scraped this page: http://web.mac.com/j3mbeck/iWeb/JohnBeckPaper_Steel/Fireplace%20Surrounds.html
The alt attribute of the first thumbnail, steel surround, contains the text "Steel has that effect where you'd least expect it". The ' character shows up as Õ when I use the method below, and the "d" is just swallowed.
data.gsub!(/\323/, '"')
require 'oniguruma'
o = Oniguruma::ORegexp.new('[^[:ascii:]]') # o = Oniguruma::ORegexp.new('[^[:ascii:]]', {:encoding => Oniguruma::ENCODING_UTF8}) chars = data.each_char{|c|chars << c} chars.collect do |c| if o.match c begin "&##{c.unpack('U*').first};" rescue ArgumentError add_log_message("Has malformed UTF-8 characters") #handling malformed UTF-8 : a huge pain and possibly future cause of problems bytes = c.each_byte{|b| bytes << b} # assumes we're handling at most, 2-byte strings. We have no way if the malformed character is # supposed to be one byte or two, but we're assuming it's 1. ["&##{bytes[0]}"] + bytes[1..-1].collect{|b|b.chr} end else c end end.flatten.join('')
Any suggestions?
Thanks! Daniel
