Will Rails be able to do this -fast- ?

Hi,

First of all I'd like to say I really love this Rails community. Everyone is very helpful and I've learned so much that I simply couldn't have learned just by reading books. I was wondering if you could help me with the following as well. My general question is: will Rails be the way to go with this?

I need an application that does server monitoring. But not general server monitoring (as in ping specific ports), it actually queries servers for information and based on that information it will act. There will be a list of servers and generally, when the end of the list is reached, it will start all over again. The list will be updated by the application, but also by user input. I need to be able to set an interval between the requests, but the application also needs to know about certain limitations. For example, there will be a maximum of queries to a specific server in a single hour as some database entries point to the same server, but with a different query. At this point I ask you - is this all possible? The application needs to run on itself for months, maybe even years in a row and be very low- maintenance.

Well, if that's possible with Rails, there's another part of the application that needs to act on specific results. For example, when something's not the way it should be, it needs to send an e-mail. The thing is, this e-mail needs to be sent -fast- ! I'm talking about milliseconds here. This is because the e-mail will be processed by another server automatically. Every millisecond counts with this. Again, I ask you: is Rails the way to go?

I'm quite sure I want to develop the front-end in Rails, but I'm unsure as to whether the backend will be just as fast in Rails as in any other programming language. Are there things to think about here? For example, I don't know how I would best send these e-mails. SMTP? Is this fast in Rails? Any faster or slower than in any other programming language? And why? I'm very inexperienced with the properties of various available programming languages and wouldn't know where to start exploring this.

And how can I make this an ongoing process? I don't want a cronjob- powered script, for example. Later on, when there's a lot to query, we might move from sending a query once every five seconds to a hundred times per second. How to do this? It could also be also time-specific. Some database entries only need to be monitored from 8AM to 5PM for example. But not all.

I hope you can help me with this, as I'm not sure where to start. If you tell me to go and learn another programming language that's faster, I will. Just be honest.

Thank you very much.

www.spiceworks.com?

I love Spiceworks, but it's not what I'm looking for here. Sorry!

And how can I make this an ongoing process? I don't want a cronjob- powered script, for example. Later on, when there's a lot to query, we might move from sending a query once every five seconds to a hundred times per second. How to do this? It could also be also time-specific. Some database entries only need to be monitored from 8AM to 5PM for example. But not all.

I hope you can help me with this, as I'm not sure where to start. If you tell me to go and learn another programming language that's faster, I will. Just be honest.

To be honest, if you're sending emails, querying remote servers etc. then you shouldn't be worrying about milliseconds - either of those two actions could take far longer than that. The smtp server you talk too could easily sit on your message for seconds or even minutes - email isn't an instant delivery protocol. Your server-monitory thing is going to be some sort of background process, you may find a single instance of that gets more done if you're using something like jruby that has better threading than normal ruby. Stuff like eventmachine is pretty cool for having a single process multiplex a lot of IO.

Fred

In a prior life I wrote a control process for the FAA that managed a distributed suite. The suite was spread over three sites, 45+ pieces of hardware, 25 communicating apps, 165+ command line arguments and provided near realtime advice to air traffic controllers. Features included near real time sensing of failed hardware and software with automated failover to redundant components. The system needed to be site configurable to support (initially) four regions.

I started by developing a language specific to the problem at hand and implemented this using lex and yacc. For infrastructure I used the Parallel Virtual Machine (PVM) from Oak Ridge National Laboratory, all coding was done in C with user application libraries available in C and C++.

Ruby and Rails are fine applications but I would not consider them suitable for any time sensitive application. You could build useful support and analysis tools and you might even be able to build a simulator, but... My guess is that in a distributed monitor and control situation you'ld be looking at sense and respond times in the multi tens of seconds.

Thanks Frederick,

The thing is, I can't choose the tools I need to work with. I'm not worrying about server response time from the servers that I will query, that's just a given and I cannot do anything about it. I also cannot worry about e-mail delivery time -after it left my machine- because that also is just a given. The e-mails will be sent to servers physically very close to the server sending out the e-mails. Just believe me, any millisecond counts. That's also the reason I will be creating this application: I will need to be able to fine-tune it later on so that it fits perfectly.

To Rick:

Do you think C and/or C++ are more capable of doing this? I have no prior experience with C, so I'll have to look into the possibilities.

To all:

Is Rails, or at least Ruby, able to do server monitoring on pre-set intervals? Forget about the speed issue for a moment, but would it be able to query a server, wait for its response (we're talking about normal HTTP-requests), store this response and go on? There will be three applications running on three servers. One is just querying -a lot- of servers and it doesn't matter whether it has to wait for the previous request to finish or not (so I guess multi-threading will not be needed, right?). It only needs to store the last response and the time of that last response. The second is querying only a few servers and there are limitations to how often it can query a specific server, but it should do a request every second (meaning that when a request takes over 1 second, it will need to start a new thread). "Normal" Ruby will probably not be able to do this, right? The last one will do many requests per second to one or two servers, also on a pre-set interval, but it also needs to send out an e-mail when the response differs from what is expected. I guess, especially for this last application, Ruby won't be able to help me. Am I right?

I'm going to buy some C books, at least for app 2 and 3, I guess :wink:

Thank you both!

Thanks Frederick,

The thing is, I can't choose the tools I need to work with. I'm not worrying about server response time from the servers that I will query, that's just a given and I cannot do anything about it. I also cannot worry about e-mail delivery time -after it left my machine- because that also is just a given. The e-mails will be sent to servers physically very close to the server sending out the e-mails. Just believe me, any millisecond counts. That's also the reason I will be creating this application: I will need to be able to fine-tune it later on so that it fits perfectly.

Just seems highly likely that any time you save will actually be lost in the noise.

Is Rails, or at least Ruby, able to do server monitoring on pre-set intervals? Forget about the speed issue for a moment, but would it be able to query a server, wait for its response (we're talking about normal HTTP-requests), store this response and go on?

Don't see why not. As you suspect this bit wouldn't actually use rails at all (maybe activerecord if you want to store the results in a database)

There will be three applications running on three servers. One is just querying -a lot- of servers and it doesn't matter whether it has to wait for the previous request to finish or not (so I guess multi-threading will not be needed, right?). It only needs to store the last response and the time of that last response. The second is querying only a few servers and there are limitations to how often it can query a specific server, but it should do a request every second (meaning that when a request takes over 1 second, it will need to start a new thread). "Normal" Ruby will probably not be able to do this, right? The last one will do many requests per second to one or two servers, also on a pre-set interval, but it also needs to send out an e-mail when the response differs from what is expected. I guess, especially for this last application, Ruby won't be able to help me. Am I right?

stuff like eventmachine does all the gnarly IO bits in C so can be pretty effective for this sort of stuff if you're worried about the performance side

Fred

Thanks Fred :slight_smile:

Just seems highly likely that any time you save will actually be lost in the noise.

If it does, it does. However, on average, an e-mail sent 1 millisecond faster will also be received 1 ms faster. I know you think it doesn't matter, but for this application, it matters. Trust me on this. Even server load isn't as important as speed.

And thanks for pointing me towards the EventMachine, I knew it was there, but I just forgot.

My point in bringing in C was that the tools you choose are those that can do the job. If you have a real time requirement (something with msecs response) you won't be happy with interpreted code exchanging messages via email. However, if what you've been given is more on the order of "use Ruby on Rails and make this as fast as possible" then you can certainly get the job done.

The real tradeoff here is that to achieve real speed you must invariably give up flexibility. Even though you could write a hard drive controller in Ruby I doubt you would be satisfied with the result.

I would recommend that you spend some time defining what you are trying to do and get requirements set for the areas of your application in which performance is critical. You should have real numbers here - or agreement with your user that "as fast as possible" is adequate.

Take a look at SNMP as a mechanism for collecting data from remote systems - you might look at MRTG for example. Quite often you can partition the time sensitive portion out from the reporting portion. You could, for example, build status logs to report out when all is running in a pre-defined "good" way and make a nice web view that's driven from the logs. You obviously need to develop a strategy for reporting and responding to the alert and alarm states that populate the "not good" space. This is the place where time can be critical.

So go for it - sounds like fun.

I have to agree with Fred - trying to optimize things to this level is almost certainly a mistake. There are far too many other factors that can randomly devour >10ms at a pass: the size of the OS timeslice (typically around 10ms), swapping pages in from disk (50ms to ?), network contention, etc.

On the other hand, if you're really serious about being fast, SMTP is doing it completely wrong. Write a real TCP client and you'll clear out a massive amount of overhead.

Unless this is for one of the high-frequency stock trading operations, in which case I encourage you to spend as much time and money on it as possible - tell them they need a custom OS or something... :wink:

--Matt Jones

My point in bringing in C was that the tools you choose are those that can do the job. If you have a real time requirement (something with msecs response) you won't be happy with interpreted code exchanging messages via email. However, if what you've been given is more on the order of "use Ruby on Rails and make this as fast as possible" then you can certainly get the job done.

What I was given was: - I'd like you to develop an app in PHP - When x happens, an e-mail needs to be sent out as fast as possible and every millisecond counts.

At that moment I mentioned Rails because I had no clue on what this app needed to do and I just love Rails and want to limit the time I spend coding PHP scripts. However, now I know the details, I think it might be important to look into other programming languages and/or frameworks. Of course, cost is an issue and it's an application for which I could probably have a v0.1 ready in under 20 hours. If I need to learn a new language, though, I might be taking alot longer. I know, that's his problem, but I want to offer a set of possibilities and it's for him to decide what's most important.

The real tradeoff here is that to achieve real speed you must invariably give up flexibility. Even though you could write a hard drive controller in Ruby I doubt you would be satisfied with the result.

I would recommend that you spend some time defining what you are trying to do and get requirements set for the areas of your application in which performance is critical. You should have real numbers here - or agreement with your user that "as fast as possible" is adequate.

The numbers aren't disclosed. We will need to develop the application, cross our fingers and hope it's fast enough. It could easily be that it's not possible and our e-mail messages arrive too late every time. That's why I need to be sure that what we develop is the best we can come up with within the budget. If C is the way to go, I will offer that possibility. However, I'm quite sure I will need twice as long in C and might have less time left for fine-tuning.

Take a look at SNMP as a mechanism for collecting data from remote systems - you might look at MRTG for example. Quite often you can partition the time sensitive portion out from the reporting portion. You could, for example, build status logs to report out when all is running in a pre-defined "good" way and make a nice web view that's driven from the logs. You obviously need to develop a strategy for reporting and responding to the alert and alarm states that populate the "not good" space. This is the place where time can be critical.

What you're saying might be very important. I know the kind of information I'm looking for, so if I can assign the highest priority to the incoming data that contains this information, that could help. The third time-critical application will actually be designed so that when the time-critical stuff comes up, it drops everything it does and sends out the e-mail.

So go for it - sounds like fun.

At least it's very interesting! It will be fun when I figure it out, - if- I figure it out :wink:

Hi, Thanks Matt. Really appreciate your help :slight_smile:

> Thanks Fred :slight_smile:

> > Just seems highly likely that any time you save will actually be lost > > in the noise.

> If it does, it does. However, on average, an e-mail sent 1 millisecond > faster will also be received 1 ms faster. I know you think it doesn't > matter, but for this application, it matters. Trust me on this. Even > server load isn't as important as speed.

I have to agree with Fred - trying to optimize things to this level is almost certainly a mistake. There are far too many other factors that can randomly devour >10ms at a pass: the size of the OS timeslice (typically around 10ms), swapping pages in from disk (50ms to ?), network contention, etc.

That doesn't take away my first point. On average, every millisecond that I can shave off, will result in the e-mail getting there faster. And again, trust me, every millisecond counts. I don't want to debate on the need of shaving these milliseconds off; it's the task I got. My question is where to start shaving.

On the other hand, if you're really serious about being fast, SMTP is doing it completely wrong. Write a real TCP client and you'll clear out a massive amount of overhead.

Great, that's useful. I'll look into that.

Unless this is for one of the high-frequency stock trading operations, in which case I encourage you to spend as much time and money on it as possible - tell them they need a custom OS or something... :wink:

You're actually on the right track, but also way off :wink:

The thing is, I -will- start work on this at some point and I -will- need to make sure it's my best effort. The budget also has some weight of course, which is why I can't get too fancy, developing a custom OS for example :wink:

Well, just remember that Rails is a web framework, and Ruby is (as has been mentioned) an interpreted language. If speed matters, then you need to go as low in the language hierarchy as possible. And usually when you talk about going low, you mean C... As I was reminded here, though, bits of Ruby gems are written in C, precisely for speed reasons :slight_smile:

And if every millisecond counts, then yes, you probably want to control everything, and that means email is probably also the wrong way to go. You'll probably want to send your own messages through your own interface. I did not test this and I can be wrong, but it may be faster to just send one TCP packet with a code which gets translated to a real message when it gets to the recipient. Again .. Speed vs. Flexibility :slight_smile:

Good luck, and remember to share what you learned.. If you can't share the code :wink:

Well, just remember that Rails is a web framework, and Ruby is (as has been mentioned) an interpreted language. If speed matters, then you need to go as low in the language hierarchy as possible. And usually when you talk about going low, you mean C... As I was reminded here, though, bits of Ruby gems are written in C, precisely for speed reasons :slight_smile:

That helps :slight_smile:

And if every millisecond counts, then yes, you probably want to control everything, and that means email is probably also the wrong way to go. You'll probably want to send your own messages through your own interface. I did not test this and I can be wrong, but it may be faster to just send one TCP packet with a code which gets translated to a real message when it gets to the recipient. Again .. Speed vs. Flexibility :slight_smile:

We don't control the recipient and we have to send an e-mail. It might change, but it won't for the forseeable future.

Good luck, and remember to share what you learned.. If you can't share the code :wink:

I will share what I can share, but if I don't use Rails, I probably won't be sharing it here :wink:

You keep saying "every millisecond counts", yet your requirement was given basically as "I need an F1 racecar, here take this bicycle and build me one" to which you responded "I'm more familiar with mopeds, I'll use that instead".

Well, I'm not more familiar with Rails. I've been using PHP for over eight years now and I'm still learning Rails. However, when he asked me whether I could do "something" in PHP, I stated that I preferred using Rails. When it got clear what needed to be done, I wasn't so sure what I would need, which is why I started this topic. What's wrong about that?

I'm working on a blog to share these kinds of things with the rest of the world. I have a feeling everyone having a crazy idea starts knocking on my door lately. I've never done the things I'm trying to do now when I was using PHP. I learn so much so fast!

Lol @ news update xD

Thanks Marnen :slight_smile:

jhaagmans wrote: >> My point in bringing in C was that the tools you choose are those that >> can do the job. If you have a real time requirement (something with >> msecs response) you won't be happy with interpreted code exchanging >> messages via email. However, if what you've been given is more on the >> order of "use Ruby on Rails and make this as fast as possible" then >> you can certainly get the job done.

> What I was given was: > - I'd like you to develop an app in PHP > - When x happens, an e-mail needs to be sent out as fast as possible > and every millisecond counts.

Why not just set up Nagios and have done with it? SNMP might be useful too...

Because I will not be doing infrastructure monitoring. I'm querying servers and applications running on servers (could even be PHP scripts) for actual information which I will use in the applications.

Anyway, if you've been told every millisecond counts, you need to know *why*. Do you?

I do. And he's not exaggerating. But again, I can't go into the details. I'm absolutely 100% sure that anyone here would agree with me that every millisecond counts. But I can't say any more about it and it's useless to discuss whether it's actually needed when I can't explain why. Just believe me.

Thanks again to all. By monday I will get back in touch with my client and I will present him with options I think are usable. If anyone has any comments on how to do this, please share them.

And the plot thickens. One more time, then - you are being asked to develop a real-time application which depends on : a) a network b) other computers c) applications on other computers d) at least one email server

And every millisecond counts? [ Hey, can I have fries with that? :slight_smile: ]

Just remember to forget about the why. The how, that's what I need help with. I'll be looking at the possibilities tomorrow and decide what I think is best.

"just remember to forget" ... I should write that one down !

jhaagmans wrote:

And the plot thickens. One more time, then - you are being asked to develop a real-time application which depends on : a) a network b) other computers c) applications on other computers d) at least one email server

And every millisecond counts? [ Hey, can I have fries with that? :slight_smile: ]

Just remember to forget about the why.

Um...no. I'm not trying to pry into your requirements, but I *am* telling you that your requirements must make sense, or else your project is doomed.

You are being asked to build an F-16 with your choice of a bicycle or a moped as engine. Oh, and it has to have a fuel efficiency of 30 MPG, and don't spend too much time developing it. Your requirements do not make sense.

The how, that's what I need help with. I'll be looking at the possibilities tomorrow and decide what I think is best.

You've been given a self-inconsistent set of requirements. Don't even bother until you can get a more feasible set of requirements.

"just remember to forget" ... I should write that one down !

I should write that down too -- as a sign that a project is in trouble.

Best,

Um...no. I'm not trying to pry into your requirements, but I *am* telling you that your requirements must make sense, or else your project is doomed.

You are being asked to build an F-16 with your choice of a bicycle or a moped as engine. Oh, and it has to have a fuel efficiency of 30 MPG, and don't spend too much time developing it. Your requirements do not make sense.

I'm not bound to using PHP, Ruby or anything. I'm just bound to a certain budget which limits my possibilities. And that will limit the speed of the application. It will be a matter of as fast as possible though, which is why I want to do this using the right tool.

The requirements make sense. At least, to me they do. He also has his reasons to believe this can be done on a low budget. And the thing is, I don't doubt it.

The goal of the application is to increase the success rate of my client's business. Say it's at 1:1000 right now. If I can get it up to 1:500 (which is a matter of speed in this case) I will have myself one happy client. However, if I can get it up to 1:400, I will have an even happier client. If I exceed the budget by more than say, 10%, I will have an angry client. So I need to increase the speed by as much as possible within the budget.

So again I ask you to stop questioning the why. If it's not possible, I will find that out and so will he. But I do want to give this an honest chance.