Threads and net/http: am I missing something?

rubyonrails-talk
21

Thank you both.

Conrad, do you want me to execute this from within BackgroundRB? I'll be back tonight and test it.

22

I haven't had a chance to test Conrad's suggestion, but I've also seen that I reach the connection limit for ActiveRecord. Is there a way to do, for example:

hosts.each do |host|   #Do all kinds of stuff to host end hosts.save

So I don't have alot of concurrent connections to my database?

23

I haven’t had a chance to test Conrad’s suggestion, but I’ve also seen

that I reach the connection limit for ActiveRecord. Is there a way to

do, for example:

hosts.each do |host|

#Do all kinds of stuff to host

end

hosts.save

So I don’t have alot of concurrent connections to my database?

You should be able to wrap it into a single transaction.

Good luck,

-Conrad

24

jhaagmans a écrit, le 11/20/2009 12:41 PM :

I haven't had a chance to test Conrad's suggestion, but I've also seen that I reach the connection limit for ActiveRecord. Is there a way to do, for example:

hosts.each do |host|   #Do all kinds of stuff to host end hosts.save

So I don't have alot of concurrent connections to my database?    Are you using threads in the block ? If not, even if you use host.save in the block and not a single commit at the end, there's only one connection to the database.

Lionel

25

Lionel Bouton a écrit, le 11/20/2009 01:27 PM :

jhaagmans a écrit, le 11/20/2009 12:41 PM :   

I haven't had a chance to test Conrad's suggestion, but I've also seen that I reach the connection limit for ActiveRecord. Is there a way to do, for example:

hosts.each do |host|   #Do all kinds of stuff to host end hosts.save

So I don't have alot of concurrent connections to my database?    Are you using threads in the block ? If not, even if you use host.save in the block and not a single commit at the end, there's only one connection to the database.

Lionel

BTW, if you use threads, you should join each thread (or try to) to have at least some control over the number of threads running concurrently.

If not, each time you loop over such code you create new threads (which probably will use new DB connections, I'm not sure how the ActiveRecord DB connection pool is used when ActiveRecord is used in a threaded environment) without any guarantee that previous ones finish.

Lionel

26

BTW, if you use threads, you should join each thread (or try to) to have at least some control over the number of threads running concurrently.

If not, each time you loop over such code you create new threads (which probably will use new DB connections, I'm not sure how the ActiveRecord DB connection pool is used when ActiveRecord is used in a threaded environment) without any guarantee that previous ones finish.

You should bracket the databasey bits with ActiveRecord::Base.connection_pool.with_connection do

end

to ensure that your connection is returned to the pool (otherwise once you have consumed the number of connections in the pool your threads will start to block until the wait for new connection timeout expires.

Fred

27

Great advice (again), Fred. I had found some other tips that didn't suit my situation, but this is great! Thanks.

@Lionel: Yes, I was using threads for every host, so I should return these connections to the pool, because some BRB processes can overlap each other and have 5 or more active connection in total.

I have time to test Conrad's suggestion tomorrow, but I'd like to thank you all, because I've made huge steps forward on understanding these issues.