How to read HTML from a password-protected page?

If it is using http authentication, you can use the format of
http://user:password@joe.com?restofurl

You can use httpclient for this kind of thing:

require 'rubygems'
require 'httpclient'

url = "http://joe.com/find?word=#{word}&lb=1&user=joe"
client = HTTPClient.new
client.debug_dev = STDOUT if $DEBUG
client.set_auth(url, 'joe', 'thePassw0rd')
resp = client.get(url)
@page = resp.content

You might have to make the first arg to .set_auth be just the URL without the query string.

Also, if the service doesn't send back an authorization request (which it sounds like it does since the browser asks you), then HTTPClient won't volunteer the credentials you've provided.

You can check things like if resp.status != 200

-Rob

Rob Biedenharn http://agileconsultingllc.com
Rob@AgileConsultingLLC.com