Mechanize GETting twice without redirect?

rubyonrails-talk
1

[Cross posted to Ruby on Rails Forum and Mechanize mailing list.]

I'm using Mechanize for page scraping (Ruby 1.9.2 / Rails 3.0.5 / Mechanize 2.0.1). I'm seeing a case where a single

  agent.get(url)

generates two HTTP GETs. Why is this happening?

The response to the first GET is a 200 (no redirect) and doesn't have any meta-refresh. I don't see why Mechanize is issuing the second GET (which happens to be failing with an EOFError with Content-Length / body length mismatch).

Details: I'm using the nifty Charles web proxy debugger to monitor browser / server interactions.

2

UPDATE: sort of solved.

The web server sets Content-Length to a incorrect value (at least for compressed replies), and Mechanize was re-trying the GET before giving up.

The temporary fix is to monkey patch Mechanize::HTTP::Agent to ignore Content-Length.

A longer-term fix would be to include a mechanism in Mechanize to allow ignoring bogus Content-Length. I've submitted a feature request to mechanize/issues.