Outputting to web = Rails or web framework the best?

Rails would be great for something like that. You could also look
into merb (merbivore.com), I enjoy merb a lot and its a little lighter
than rails. There is also Camping which is even lighter than both.

If you are going the rails route, you may want to look into
backgroundrb to push your workers extracting those pages to a
background process. You'd then use javascript to pull in the
results. That is because rails will lock on your mongrel instance, so
if there is a delay, others may be hung up trying to access your page.

If you aren't using activerecord, you can safely turn off the mutex
lock with merb, by starting it with -X off. I'm doing that right now
to run similar things, like snmpwalks or extracting external pages and
inspect them with net/http or open-uri and hpricot.

However, you may even want to simply create a Mongrel handler for
this, if its that simple.

Check this one out:

require 'rubygems'
require 'mongrel'

class TimeHandler < Mongrel::HttpHandler
  def process(request,response)
    response.start(200) do |headers, output|
      headers["Content-Type"] = "text/plain"
      output.write(Time.now)
    end
  end
end

server = Mongrel::HttpServer.new("0.0.0.0", "2222")
server.register("/time", TimeHandler.new)
server.run.join

Run that and then go to http://localhost:2222/time

That should get you started.