Threads and net/http: am I missing something?

rubyonrails-talk
#1

Hi,

I have a backgroundrb worker that gets triggered every second. When
it's triggered, it's supposed to make 2 - 15 http-requests using
Net::HTTP. My idea was to put every execution into a thread so the
next execution doesn't have to wait for the last one. So basically:

def http_requests
  hosts.each do |host|
    Thread.new do
      begin
        client = Net::HTTP.start(host)
      rescue
        #store host as inactive
      ensure
        client.finish if client.active?
      end
    end
  end
end

Of course that's not all it does, but I hope you understand what I'm
trying to do here.

The thing is: this doesn't get done once a second. It appears that
every HTTP-request is waiting for the last one to complete, which
clots up Rails very fast!

My question is: why is this? Does this have anything to do with Ruby
not being threadsafe (I doubt it, because that just means threads
aren't executed as precisely as with jRuby, right?) or is Net::HTTP
not able to make requests while another Net::HTTP request is still
running? And what to do?

I hope you can help.

#2

Update: I've tried doing it using EventMachine, which won't work
either:

EM.run do
  make_request(host)
end

def make_request
  begin
    client = EventMachine::HttpRequest.new(host).get
    host.set_active
  rescue
    host.set_inactive
  ensure
    EM.stop
  end
end

Now it's only executing the set_active or set_inactive methods every
five seconds, even though I run this every second for 5 hosts. It
should query 5 hosts per second. What am I missing? Is this EM-
related? Should I have EventMachine check for responses more often?

#3

Hi,

I have a backgroundrb worker that gets triggered every second. When

it’s triggered, it’s supposed to make 2 - 15 http-requests using

Net::HTTP. My idea was to put every execution into a thread so the

next execution doesn’t have to wait for the last one. So basically:

def http_requests

hosts.each do |host|

Thread.new do

  begin

    client = Net::HTTP.start(host)

  rescue

    #store host as inactive

  ensure

    client.finish if client.active?

  end

end

end

end

Of course that’s not all it does, but I hope you understand what I’m

trying to do here.

The thing is: this doesn’t get done once a second. It appears that

every HTTP-request is waiting for the last one to complete, which

clots up Rails very fast!

My question is: why is this? Does this have anything to do with Ruby

not being threadsafe (I doubt it, because that just means threads

aren’t executed as precisely as with jRuby, right?) or is Net::HTTP

not able to make requests while another Net::HTTP request is still

running? And what to do?

I hope you can help.

The Global Interpreter Lock (GIL) prevents threads from executing in

parallel when using Ruby 1.8.6 aka MRI, 1.8.7, and 1.9.1 aka YARV.

However, JRuby 1.3.x/1.4.x, MacRuby 0.5 Beta 2, Maglev and several

other upcoming Ruby VMs are not constrained by the GIL. Thus, they

can execute threads in parallel.

Good luck,

-Conrad

#4

Emm just because the threads aren't all executing *simultaneously*
doesn't that they aren't running in parallel (due to all the thread
switching etc).

Regardless, I can't seem to reproduce the OPs behviour:

require 'net/http'

def hosts
  %w[rubyforge.org www.scala-lang.org www.google.com www.gamefaqs.com
allrecipes.com m2k2.taigaforum.com youtube.com gitorious.org
everything2.com]
end

def http_requests
  hosts.each do |host|
    Thread.new do
      begin
        puts "fetching host #{host}"
        client = Net::HTTP.start(host)
      rescue e
        #store host as inactive
      ensure
        puts "finished with host #{host}"
        client.finish if client.active?
      end
    end
  end
end

irb(main):001:0> http_requests
fetching host rubyforge.orgfetching host www.scala-lang.orgfetching
host www.google.com
fetching host www.gamefaqs.com
fetching host allrecipes.com
fetching host m2k2.taigaforum.com
finished with host rubyforge.orgfetching host youtube.com
fetching host gitorious.org
finished with host www.google.com
fetching host everything2.com
=> ["rubyforge.org", "www.scala-lang.org", "www.google.com",
"www.gamefaqs.com", "allrecipes.com", "m2k2.taigaforum.com",
"youtube.com", "gitorious.org", "everything2.com"]
irb(main):002:0>

finished with host www.scala-lang.org
finished with host m2k2.taigaforum.comfinished with host www.gamefaqs.com
finished with host allrecipes.com

finished with host youtube.com
finished with host everything2.com
finished with host gitorious.org

Am I missing something or misunderstanding the question?

#5

Also "threadsafe" doesn't have anything to do with language itself (or
even its implementations) but rather if code operates correctly and
predictably when run in parallel. But blah terminology.

#6

pharrington a écrit, le 11/18/2009 07:23 PM :

Emm just because the threads aren't all executing *simultaneously*
doesn't that they aren't running in parallel (due to all the thread
switching etc).

Regardless, I can't seem to reproduce the OPs behviour:

I'm too lazy to check the details, but I'd look at the implementation
details of BackroundRB and EventMachine. I suspect the way they use
select calls may interract badly with Net::HTTP.

For example, last time I checked if you wrap an HTTP get in a timeout
block, the timeout doesn't work : the internal Net::HTTP timeouts take
precedence and disable the global timeout.
Timeout uses a thread which calls sleep, which is implemented with
select IIRC...

Lionel

#7

Hi,

I have a backgroundrb worker that gets triggered every second. When

it’s triggered, it’s supposed to make 2 - 15 http-requests using

Net::HTTP. My idea was to put every execution into a thread so the

next execution doesn’t have to wait for the last one. So basically:

def http_requests

hosts.each do |host|

Thread.new do

 begin

   client = Net::HTTP.start(host)

 rescue

   #store host as inactive

 ensure

   client.finish if client.active?

 end

end

end

end

Of course that’s not all it does, but I hope you understand what I’m

trying to do here.

The thing is: this doesn’t get done once a second. It appears that

every HTTP-request is waiting for the last one to complete, which

clots up Rails very fast!

My question is: why is this? Does this have anything to do with Ruby

not being threadsafe (I doubt it, because that just means threads

aren’t executed as precisely as with jRuby, right?) or is Net::HTTP

not able to make requests while another Net::HTTP request is still

running? And what to do?

I hope you can help.

The Global Interpreter Lock (GIL) prevents threads from executing in

parallel when using Ruby 1.8.6 aka MRI, 1.8.7, and 1.9.1 aka YARV.

However, JRuby 1.3.x/1.4.x, MacRuby 0.5 Beta 2, Maglev and several

other upcoming Ruby VMs are not constrained by the GIL. Thus, they

can execute threads in parallel.

Good luck,

-Conrad

Emm just because the threads aren’t all executing simultaneously

doesn’t that they aren’t running in parallel (due to all the thread

switching etc).

Each thread must acquire the lock before it can execute. Thus, it operates

similar to a queue data structure (i.e. first in first out (FIFO)) and this is how

it work today in regards to Ruby 1.8.6, 1.8.7, and 1.9.1. I know the C implementation

of the Ruby VM very well.

-Conrad

#8

Wow, thanks for all your help, greatly appreciated.

The Global Interpreter Lock (GIL) prevents threads from executing in
parallel when using Ruby 1.8.6 aka MRI, 1.8.7, and 1.9.1 aka YARV.
However, JRuby 1.3.x/1.4.x, MacRuby 0.5 Beta 2, Maglev and several
other upcoming Ruby VMs are not constrained by the GIL. Thus, they
can execute threads in parallel.

The reason why we didn't choose jRuby was because it uses too much
memory to be able to run this on a VPS. Is there any documentation
available on using jRuby on a low-memory (<256MB) system? I've looked
for it, but couldn't find it. Maybe there's an alternative workaround
for the GIL? Our application uses up quite alot of memory, so when
presented with the jRuby vs. Ruby (EE) question, I thought it was a
choice between thread safety and memory usage, so I chose the latter.
I didn't know there was more to think about.

Regardless, I can't seem to reproduce the OPs behviour:

Which Ruby implementation are you using? I'm very sure every thread in
my piece of code is waiting for the other thread to finish, because I
log the time at which the data is saved. Most of the time there's 10 -
40 seconds between them, even though the backgroundrb process should
save at least one object every second.

#9

Each thread must acquire the lock before it can execute. Thus, it operates
similar to a queue data structure (i.e. first in first out (FIFO)) and this
is how
it work today in regards to Ruby 1.8.6, 1.8.7, and 1.9.1. I know the C
implementation
of the Ruby VM very well.

-Conrad

The C code will acquire the GIL yes, and then release it when its done
its bit of business. This will happen any number of times within a
given function. So yes while the first thread created is the first to
run its bit of code, in no way does that mean its the first thread to
finish, nor does it stop the interpreter from switching control to
another than when the lock is given up in the middle of execution.
Saying Ruby threads don't run in parallel is even less true than
saying coroutines aren't a form of parallelism.

The reason why we didn't choose jRuby was because it uses too much
memory to be able to run this on a VPS. Is there any documentation
available on using jRuby on a low-memory (<256MB) system? I've looked
for it, but couldn't find it. Maybe there's an alternative workaround
for the GIL? Our application uses up quite alot of memory, so when
presented with the jRuby vs. Ruby (EE) question, I thought it was a
choice between thread safety and memory usage, so I chose the latter.
I didn't know there was more to think about.

> Regardless, I can't seem to reproduce the OPs behviour:

Which Ruby implementation are you using? I'm very sure every thread in
my piece of code is waiting for the other thread to finish, because I
log the time at which the data is saved. Most of the time there's 10 -
40 seconds between them, even though the backgroundrb process should
save at least one object every second.

xeno@Clover:~/projects$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Threading's not the issue. Try looking into Lionel's suggestion;
perhaps things will work as expected if you switch to delayed_job
instead of BackgroundRB, or posssssibly even just a different HTTP
client like curb. Or perhaps the problem's not related to the HTTP
fetching at all and we don't have the whole story.

#10

> The reason why we didn't choose jRuby was because it uses too much
> memory to be able to run this on a VPS. Is there any documentation
> available on using jRuby on a low-memory (<256MB) system? I've looked
> for it, but couldn't find it. Maybe there's an alternative workaround
> for the GIL? Our application uses up quite alot of memory, so when
> presented with the jRuby vs. Ruby (EE) question, I thought it was a
> choice between thread safety and memory usage, so I chose the latter.
> I didn't know there was more to think about.

> > Regardless, I can't seem to reproduce the OPs behviour:

> Which Ruby implementation are you using? I'm very sure every thread in
> my piece of code is waiting for the other thread to finish, because I
> log the time at which the data is saved. Most of the time there's 10 -
> 40 seconds between them, even though the backgroundrb process should
> save at least one object every second.

xeno@Clover:~/projects$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Threading's not the issue. Try looking into Lionel's suggestion;
perhaps things will work as expected if you switch to delayed_job
instead of BackgroundRB, or posssssibly even just a different HTTP
client like curb. Or perhaps the problem's not related to the HTTP
fetching at all and we don't have the whole story.

If the problem is in BackgroundRB I'm screwed. This whole system
depends on BackgroundRB. It's not just a long-running task, most tasks
will run daily for years and years to come. I'll check whether this is
related to BackgroundRB and report back.

#11

Well, BackgroundRB does have a problem because workers can't overlap,
so a worker is put in the queue when it's started, which is a bit of a
pain because some HTTP-requests take longer than others. But look at
this:

  def schedule_queries
    i =
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,35]
    i.each do |j|
      Thread.new do
        logger.info "Hello #{j}"
        sleep 2
        logger.info "Hello Again #{j}"
      end
      sleep 1
    end
  end

This is executed every 10 seconds and my log shows this:

Hello 1
Hello 2
Hello Again 1
Hello 3
Hello Again 2
Hello 4
Hello Again 3
Hello 5
Hello Again 4
Hello 6
Hello Again 5

(...)

Hello Again 23
Hello 35
Hello Again 24
Hello Again 35
Hello 1
Hello 2
Hello Again 1
Hello 3

The expected behaviour is for the second "Hello 1" to appear after the
first "Hello 10", but it doesn't. However, that's not a real problem,
because I can easily work with bigger collections. However, when I
implement the script you presented earlier, it works! I don't know
why, I certainly don't know how, but it does.

Could this have anything to do with ActiveRecord? Every time I call
set_active, I do a host.save. Could that be the problem? I'll work on
this some more in the morning.

#12

Each thread must acquire the lock before it can execute. Thus, it operates

similar to a queue data structure (i.e. first in first out (FIFO)) and this

is how

it work today in regards to Ruby 1.8.6, 1.8.7, and 1.9.1. I know the C

implementation

of the Ruby VM very well.

-Conrad

The C code will acquire the GIL yes, and then release it when its done

its bit of business. This will happen any number of times within a

given function. So yes while the first thread created is the first to

run its bit of code, in no way does that mean its the first thread to

finish, nor does it stop the interpreter from switching control to

another than when the lock is given up in the middle of execution.

Saying Ruby threads don’t run in parallel is even less true than

saying coroutines aren’t a form of parallelism.

Actually, the C doesn’t require a GIL because it’s being executed outside the VM

within its own process. Thus, one can simulate very good parallel execution using

something like BackgroundRB because it’s implemented as a Ruby native-extension.

-Conrad

#13

I'm a little further on this. I've started logging the process instead
of writing to ActiveRecord. This is my code:

def schedule_queries
  t = Time.now
  hosts = get_hosts(30)
  logger.info "Starting request for #{hosts.count} domains at #{t}"
  domains.each do |domain|
    Thread.new do
      begin
        logger.info "Making request for #{host.identifier} at #
{Time.now}"
        client = Net::HTTP.start(host.url)
        #set_active(domain)
        logger.info "Finished request for #{host.identifier} at #
{Time.now}"
      rescue
        #set_inactive(domain)
        logger.info "Error in request for #{host.identifier} at #
{Time.now}"
      ensure
        client.finish if client.active? if !client.nil?
      end
    end
  end
end

The log shows this:

Starting request for 30 domains at Thu Nov 19 11:50:01 +0100 2009
Making request for dym at Thu Nov 19 11:50:01 +0100 2009
Making request for nsn at Thu Nov 19 11:50:11 +0100 2009
Finished request for dym at Thu Nov 19 11:50:21 +0100 2009
Making request for ren at Thu Nov 19 11:50:21 +0100 2009
Finished request for nsn at Thu Nov 19 11:50:31 +0100 2009
Making request for ixf at Thu Nov 19 11:50:31 +0100 2009
Finished request for ren at Thu Nov 19 11:50:41 +0100 2009
Making request for phw at Thu Nov 19 11:50:41 +0100 2009
Finished request for ixf at Thu Nov 19 11:50:51 +0100 2009
Making request for frk at Thu Nov 19 11:50:51 +0100 2009
Finished request for phw at Thu Nov 19 11:51:01 +0100 2009
Making request for gyt at Thu Nov 19 11:51:01 +0100 2009
Finished request for frk at Thu Nov 19 11:51:11 +0100 2009
Making request for nlb at Thu Nov 19 11:51:11 +0100 2009
Finished request for gyt at Thu Nov 19 11:51:21 +0100 2009
Making request for tdz at Thu Nov 19 11:51:21 +0100 2009
Error in request for tdz at Thu Nov 19 11:51:39 +0100 2009
Finished request for nlb at Thu Nov 19 11:51:39 +0100 2009

As you can see, it does do -some- threading, but it finishes requests
only once every 10 seconds or so. What am I doing wrong? pharrington's
example works for me, but this one doesn't.

#14

Some things can block the entire ruby VM - you may be falling foul of
one of them. In particular, domain name resolution can do that (there
is a pure ruby dns resolver which doesn't have that caveat)

Fred

#15

Hi Fred,

Thanks for your response. I hope you can answer three questions about
this.

- Why is it that pharrington's example did work, even though the hosts
he used (I copied them) were never resolved before on my server? I
should have had the same problem, right?

- I've used net-dns before because what I really want to do is check a
domain for existence in the DNS-records of a certain server. However,
the problem with this is that because of the TTL, when I query a
domain every hour, a domain that has been deleted from the DNS doesn't
really get deleted from the "visible" records for 4 - 24 hours. That's
something I really need to work around, do you have an idea how I can
get that to work?

- Why is it that most of the logged messages are 10 seconds apart?
That should tell me something, but I'm unsure what.

Thanks again.

Jaap

#16

Hi Fred,

Thanks for your response. I hope you can answer three questions about
this.

- Why is it that pharrington's example did work, even though the hosts
he used (I copied them) were never resolved before on my server? I
should have had the same problem, right?

maybe, maybe not - not sure what varies between different dns lookups.

- I've used net-dns before because what I really want to do is check a
domain for existence in the DNS-records of a certain server. However,
the problem with this is that because of the TTL, when I query a
domain every hour, a domain that has been deleted from the DNS doesn't
really get deleted from the "visible" records for 4 - 24 hours. That's
something I really need to work around, do you have an idea how I can
get that to work?

Are you really working around it by using net/http ?

- Why is it that most of the logged messages are 10 seconds apart?
That should tell me something, but I'm unsure what.

Is this still inside backgroundrb or have you managed to reproduce
this outside background rb ?

Fred

#17

Hi Fred,

Thanks for your response. I hope you can answer three questions about

this.

  • Why is it that pharrington’s example did work, even though the hosts

he used (I copied them) were never resolved before on my server? I

should have had the same problem, right?

  • I’ve used net-dns before because what I really want to do is check a

domain for existence in the DNS-records of a certain server. However,

the problem with this is that because of the TTL, when I query a

domain every hour, a domain that has been deleted from the DNS doesn’t

really get deleted from the “visible” records for 4 - 24 hours. That’s

something I really need to work around, do you have an idea how I can

get that to work?

  • Why is it that most of the logged messages are 10 seconds apart?

That should tell me something, but I’m unsure what.

Thanks again.

Jaap

I’m a little further on this. I’ve started logging the process instead

of writing to ActiveRecord. This is my code:

As you can see, it does do -some- threading, but it finishes requests

only once every 10 seconds or so. What am I doing wrong? pharrington’s

example works for me, but this one doesn’t.

Some things can block the entire ruby VM - you may be falling foul of

one of them. In particular, domain name resolution can do that (there

is a pure ruby dns resolver which doesn’t have that caveat)

Fred

Jaap,which Ruby VM you’re using? Also, are you still using the BackgroundDRB?

-Conrad

#18

Hi Fred,

On Nov 19, 1:22 pm, jhaagmans <jaap.haagm...@gmail.com> wrote:> Hi Fred,

> Thanks for your response. I hope you can answer three questions about
> this.

> - Why is it that pharrington's example did work, even though the hosts
> he used (I copied them) were never resolved before on my server? I
> should have had the same problem, right?

maybe, maybe not - not sure what varies between different dns lookups.

Me neither. That's why I was wondering.

> - I've used net-dns before because what I really want to do is check a
> domain for existence in the DNS-records of a certain server. However,
> the problem with this is that because of the TTL, when I query a
> domain every hour, a domain that has been deleted from the DNS doesn't
> really get deleted from the "visible" records for 4 - 24 hours. That's
> something I really need to work around, do you have an idea how I can
> get that to work?

Are you really working around it by using net/http ?

Good point. The answer is no. I thought I'd work around it because
with a full HTTP-request you'd not only query the DNS, you'd also
query the webserver, but if someone doesn't actually delete the files
from the server, you'd still get a 200-response.

Now I need to work around that as well and I doubt it's possible as we
don't actually control the DNS servers we use. I can't think of a
workaround for this.

> - Why is it that most of the logged messages are 10 seconds apart?
> That should tell me something, but I'm unsure what.

Is this still inside backgroundrb or have you managed to reproduce
this outside background rb ?

You're right, it's still inside backgroundrb. However, I ran
pharrington's example from BRB as well.

@Conrad:

[jaap@server06 ~]$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-linux], MBARI 0x8770,
Ruby Enterprise Edition 20090928

And yes, I'm still using BRB, I really can't think of a way to avoid
using BRB. I need to query a few thousand hosts every hour, every day.

I really regret choosing Ruby/Rails for this particular application at
this point.

#19

jhaagmans a écrit, le 11/19/2009 03:34 PM :

And yes, I'm still using BRB, I really can't think of a way to avoid
using BRB. I need to query a few thousand hosts every hour, every day.

This translates to a few hosts each second. If I had this kind of load I
wouldn't use a background job scheduler but a queue manager, several
processes picking the requests from the queue and a custom-built
scheduler. In fact I'm doing this for several projects and I use
ActiveMessaging.

I really regret choosing Ruby/Rails for this particular application at
this point.
  
Ruby/Rails has not much to do with your problems. Choosing the right
tool for the job is the issue.
I've a server running a web spider hand-coded in Ruby using the tools I
described above and it successfully make 10s of thousands of HTTP
queries per hour. The only limit I hit is the RAM available as I use
several Ruby processes (which, by the way, internally use threads to
handle simultaneous HTTP HEAD/GET requests efficiently). If I had say
32G i could probably hit 100k requests per hour. Given I use a queue
manager, I could also add a second server and run queue processors on it
to get more capacity...

Lionel

#20

Hi Fred,

On Nov 19, 1:22 pm, jhaagmans jaap.haagm...@gmail.com wrote:> Hi Fred,

Thanks for your response. I hope you can answer three questions about

this.

  • Why is it that pharrington’s example did work, even though the hosts

he used (I copied them) were never resolved before on my server? I

should have had the same problem, right?

maybe, maybe not - not sure what varies between different dns lookups.

Me neither. That’s why I was wondering.

  • I’ve used net-dns before because what I really want to do is check a

domain for existence in the DNS-records of a certain server. However,

the problem with this is that because of the TTL, when I query a

domain every hour, a domain that has been deleted from the DNS doesn’t

really get deleted from the “visible” records for 4 - 24 hours. That’s

something I really need to work around, do you have an idea how I can

get that to work?

Are you really working around it by using net/http ?

Good point. The answer is no. I thought I’d work around it because

with a full HTTP-request you’d not only query the DNS, you’d also

query the webserver, but if someone doesn’t actually delete the files

from the server, you’d still get a 200-response.

Now I need to work around that as well and I doubt it’s possible as we

don’t actually control the DNS servers we use. I can’t think of a

workaround for this.

  • Why is it that most of the logged messages are 10 seconds apart?

That should tell me something, but I’m unsure what.

Is this still inside backgroundrb or have you managed to reproduce

this outside background rb ?

You’re right, it’s still inside backgroundrb. However, I ran

pharrington’s example from BRB as well.

@Conrad:

[jaap@server06 ~]$ ruby -v

ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-linux], MBARI 0x8770,

Ruby Enterprise Edition 20090928

And yes, I’m still using BRB, I really can’t think of a way to avoid

using BRB. I need to query a few thousand hosts every hour, every day.

I really regret choosing Ruby/Rails for this particular application at

this point.

Jaap, please tell if the following works for you:

require ‘net/http’

module Enumerable

def concurrently

map{ |item| Thread.new { yield item }}.each{ |t| t.join }

end

end

def hosts

%w[rubyforge.org www.scala-lang.org www.google.com www.gamefaqs.com

allrecipes.com m2k2.taigaforum.com youtube.com gitorious.org

everything2.com]

end

hosts.concurrently do |host|

begin

puts “\nfetching host #{host} - #{Time.now}\n”

client = Net::HTTP.start(host)

rescue e

#store host as inactive

ensure

puts “\nfinished with host #{host} - #{Time.now}\n”

client.finish if client.active?

end

end

-Conrad