Mechanize GETting twice without redirect?

[Cross posted to Ruby on Rails Forum and Mechanize mailing list.]

I'm using Mechanize for page scraping (Ruby 1.9.2 / Rails 3.0.5 /
Mechanize 2.0.1). I'm seeing a case where a single


generates two HTTP GETs. Why is this happening?

The response to the first GET is a 200 (no redirect) and doesn't have
any meta-refresh. I don't see why Mechanize is issuing the second GET
(which happens to be failing with an EOFError with Content-Length / body
length mismatch).

Details: I'm using the nifty Charles web proxy debugger to monitor
browser / server interactions.

UPDATE: sort of solved.

The web server sets Content-Length to a incorrect value (at least for
compressed replies), and Mechanize was re-trying the GET before giving

The temporary fix is to monkey patch Mechanize::HTTP::Agent to ignore

A longer-term fix would be to include a mechanism in Mechanize to allow
ignoring bogus Content-Length. I've submitted a feature request to