Rails would be great for something like that. You could also look into merb (merbivore.com), I enjoy merb a lot and its a little lighter than rails. There is also Camping which is even lighter than both.
If you are going the rails route, you may want to look into backgroundrb to push your workers extracting those pages to a background process. You'd then use javascript to pull in the results. That is because rails will lock on your mongrel instance, so if there is a delay, others may be hung up trying to access your page.
If you aren't using activerecord, you can safely turn off the mutex lock with merb, by starting it with -X off. I'm doing that right now to run similar things, like snmpwalks or extracting external pages and inspect them with net/http or open-uri and hpricot.
However, you may even want to simply create a Mongrel handler for this, if its that simple.
Check this one out:
require 'rubygems' require 'mongrel'
class TimeHandler < Mongrel::HttpHandler def process(request,response) response.start(200) do |headers, output| headers["Content-Type"] = "text/plain" output.write(Time.now) end end end
server = Mongrel::HttpServer.new("0.0.0.0", "2222") server.register("/time", TimeHandler.new) server.run.join
Run that and then go to http://localhost:2222/time
That should get you started.
