Testing on real data

rubyonrails-talk
1

I'm still a relative newbie to RoR, but I'm finding it a very powerful tool. We have a couple of RoR applications up and in production, and we're making good use of the automatic testing to make sure they're bug free.

What I would like to do is to test using the MySQL data our users have entered. Either using that data to create fixtures, or copying the production db into the test db on every test run. With the test db recreated on every run, obviously a standard outside dump won't work.

This would be a far more comprehensive test than the few fixtures we've manually created. Is there an easy way to do this?

Thanks in advance, -Brian

2

Brian Mcardle wrote:

I'm still a relative newbie to RoR, but I'm finding it a very powerful tool. We have a couple of RoR applications up and in production, and we're making good use of the automatic testing to make sure they're bug free.

What I would like to do is to test using the MySQL data our users have entered. Either using that data to create fixtures, or copying the production db into the test db on every test run. With the test db recreated on every run, obviously a standard outside dump won't work.

This would be a far more comprehensive test than the few fixtures we've manually created. Is there an easy way to do this?

You really don't want to copy the whole production DB for every run. You might want to take a snapshot of it and use it as a set of production data.

Note also that that is the only use case in which I would *ever* recommend touching Rails' fixtures. For routine testing such as you've been doing, you will be a lot happier if you replace your fixtures with factories.

Thanks in advance, -Brian

Best,

3

Brian Mcardle wrote:

I'm still a relative newbie to RoR, but I'm finding it a very powerful tool. We have a couple of RoR applications up and in production, and we're making good use of the automatic testing to make sure they're bug free.

What I would like to do is to test using the MySQL data our users have entered. Either using that data to create fixtures, or copying the production db into the test db on every test run. With the test db recreated on every run, obviously a standard outside dump won't work.

This would be a far more comprehensive test than the few fixtures we've manually created. Is there an easy way to do this?

Personally, I disagree with this assessment. If you're finding problems with the data users are entering you've already failed them, and possibly lost their business.

I believe that it is our responsibility, as developers, to consider all possible conditions that could make something fail. Obviously this is an impossible task. There will be some edge cases that get overlooked.

The reason I disagree with your proposed approach is that it could very easily lead to developer complacency. Plan for zero failures by writing examples to test every edge case you can imagine. Some will be missed, however, I would deal with these on a case-by-case basis by ensuring you back your testing with excellent failure notification.

Also, plan for a comprehensive beta testing phase. For any notifications you receive, write an example to cover whatever was missed. If you fulfilled your part of the "contract" as a developer, by creating exhaustive tests, then these reports should be relatively rare by the time you go into production. If you still have a lot of failures in production, then you simply haven't done your job.

P.S. This response is very much an opinion, and you know what opinions are often compared with. And, I really don't mean to be one.

4

Marnen Laibow-Koser wrote:

You really don't want to copy the whole production DB for every run. You might want to take a snapshot of it and use it as a set of production data.

True - that would be my intention. Probably every month take a new snapshot, something along those lines.

Note also that that is the only use case in which I would *ever* recommend touching Rails' fixtures. For routine testing such as you've been doing, you will be a lot happier if you replace your fixtures with factories.

I'll have a look into factories and see if they'll meet my needs.

Thanks for that Marnen!

5

Robert Walker wrote:

Brian Mcardle wrote:

I'm still a relative newbie to RoR, but I'm finding it a very powerful tool. We have a couple of RoR applications up and in production, and we're making good use of the automatic testing to make sure they're bug free.

What I would like to do is to test using the MySQL data our users have entered. Either using that data to create fixtures, or copying the production db into the test db on every test run. With the test db recreated on every run, obviously a standard outside dump won't work.

This would be a far more comprehensive test than the few fixtures we've manually created. Is there an easy way to do this?

Personally, I disagree with this assessment. If you're finding problems with the data users are entering you've already failed them, and possibly lost their business.

I believe that it is our responsibility, as developers, to consider all possible conditions that could make something fail. Obviously this is an impossible task. There will be some edge cases that get overlooked.

The reason I disagree with your proposed approach is that it could very easily lead to developer complacency. Plan for zero failures by writing examples to test every edge case you can imagine. Some will be missed, however, I would deal with these on a case-by-case basis by ensuring you back your testing with excellent failure notification.

Also, plan for a comprehensive beta testing phase. For any notifications you receive, write an example to cover whatever was missed. If you fulfilled your part of the "contract" as a developer, by creating exhaustive tests, then these reports should be relatively rare by the time you go into production. If you still have a lot of failures in production, then you simply haven't done your job.

P.S. This response is very much an opinion, and you know what opinions are often compared with. And, I really don't mean to be one.

Thanks for the thoughts. I certainly agree with the general thrust of your statement - that there should be a thoroughly complete battery of tests on an application (if I understand you correctly).

But as you point out, this is impossible. We'll say I never thought of testing for non-English characters in certain strings. All my fixtures are English, all my tests pass. One of my users creates a record with a 'á' character... maybe nothing breaks. Maybe on a later upgrade, I add a new page, and that breaks.

If I had a test that used the real data, including the 'á', I would've spotted this before deploying to production, and saved myself embarrassment. As it is, every failure gets covered by new tests or monitoring.

I don't imagine these errors (those caught by db data tests) to occur more than once in a blue moon, but I believe testing with more than my manual fixtures will help me create a application with less bugs, and surely that's always a good thing?

PS I do sincerely thank you for your opinion. RoR represents a new way of thinking, and of development for me. It's been a year, but some of it is still a little ...alien to me.

6

Any one of these might help :slight_smile:

<http://nubyonrails.com/articles/dump-or-slurp-yaml-reference-data&gt; <http://www.railslodge.com/plugins/534-export-fixtures&gt; <http://www.railslodge.com/plugins/830-yaml-db&gt;

FWIW,

7

Brian Mcardle wrote:

But as you point out, this is impossible. We'll say I never thought of testing for non-English characters in certain strings. All my fixtures are English, all my tests pass. One of my users creates a record with a 'á' character... maybe nothing breaks. Maybe on a later upgrade, I add a new page, and that breaks.

I was actually speaking in generalities. However, I still don't think scenarios like you mentioned are worth the overhead of testing against production data, which will be high. And, even then, you're still going to have scenarios that testing against live data won't catch. In either case you're going to discover the problem after it has happened.

You're example illustrates this rather nicely. It would be easy to overlook accented character handling if you not expecting them. Whether you test against live data or not, you would not have prevented the problem from happening. The error would have already had to have happened. If this caused an error, and the exception notification was in place to report the error, then adding a test (or spec) to prevent this issue once know has the same effect as testing against live data without the huge overhead.

That being said if the facilities were not in place to begin with, then doing a "one-time" test against the data you have received up to this point might be useful. Just not something you want to rely upon long-term.

8

Hassan Schroeder wrote:

What I would like to do is to test using the MySQL data our users have entered. Either using that data to create fixtures

This would be a far more comprehensive test than the few fixtures we've manually created. Is there an easy way to do this?

Any one of these might help :slight_smile:

<http://nubyonrails.com/articles/dump-or-slurp-yaml-reference-data&gt; <http://www.railslodge.com/plugins/534-export-fixtures&gt; <http://www.railslodge.com/plugins/830-yaml-db&gt;

Hassan, those are excellent links! Thanks a bunch!

9

Robert Walker wrote:

Brian Mcardle wrote:

But as you point out, this is impossible. We'll say I never thought of testing for non-English characters in certain strings. All my fixtures are English, all my tests pass. One of my users creates a record with a 'á' character... maybe nothing breaks. Maybe on a later upgrade, I add a new page, and that breaks.

I was actually speaking in generalities. However, I still don't think scenarios like you mentioned are worth the overhead of testing against production data, which will be high. And, even then, you're still going to have scenarios that testing against live data won't catch. In either case you're going to discover the problem after it has happened.

You're example illustrates this rather nicely. It would be easy to overlook accented character handling if you not expecting them. Whether you test against live data or not, you would not have prevented the problem from happening. The error would have already had to have happened. If this caused an error, and the exception notification was in place to report the error, then adding a test (or spec) to prevent this issue once know has the same effect as testing against live data without the huge overhead.

Ah, a flaw in my logic. :slight_smile: You're correct there. There is still the case where a new test might be introduced, which the accented characters would break, but it's even rarer than what I was thinking of.

That being said if the facilities were not in place to begin with, then doing a "one-time" test against the data you have received up to this point might be useful. Just not something you want to rely upon long-term.

This is more what I'm aiming for. The application has been live for 8 months now, and we have a significant amount of data. I'd love to test on it and see if it breaks, then update our fixtures to take the edge cases into account. After the "one-time", it only has to be run, maybe every month, or two months, but I doubt it will make much difference after the one time.

Which is different from my original intention of using it in every test run. Thanks for your input Robert!

10

Brian Mcardle wrote:

This is more what I'm aiming for. The application has been live for 8 months now, and we have a significant amount of data. I'd love to test on it and see if it breaks, then update our fixtures to take the edge cases into account. After the "one-time", it only has to be run, maybe every month, or two months, but I doubt it will make much difference after the one time.

Which is different from my original intention of using it in every test run. Thanks for your input Robert!

One great thing about this is that we have this wonderful scripting language called Ruby. Ruby is excellent for writing scripts to automate things, as it turns out.

I'd suggest creating yourself a set of rake tasks that dumps your data out into Yaml fixture files. Then you can create tests (or specs) to run though all that data. Granted it will be very slow, but for a one-time, or not-very-often run, it might be reasonable.

11

Robert Walker wrote: [...]

I'd suggest creating yourself a set of rake tasks that dumps your data out into Yaml fixture files. Then you can create tests (or specs) to run though all that data. Granted it will be very slow, but for a one-time, or not-very-often run, it might be reasonable.

This mostly already exists. Check out yaml_db. You may need to write a little glue code.

Best,

12

Hi all,

I'm just wondering what is the best practice in rendering a page. My case is i have 3 separated forms inside 1 page, each with different functionality. Right now, i divided them into 3 partials (let's say partials a,b, and c) , and render them by using render :partials at the index page.

Partial c is constantly refreshed by using periodically_call_remote. Suppose I want to "update" the data in partial a due to user action, is it possible to do that without re-rendering the entire page?

regards, Arga

13

Yes if you make the user action an ajax call.

Colin

14

Is there any Rails way how to perform this user-based ajax call?

The periodically_call_remote is ok, but it is a periodically calls.

regards,

Arga

15

Is there any Rails way how to perform this user-based ajax call?

The periodically_call_remote is ok, but it is a periodically calls.

link_to_remote is what you want. Have a look for tutorials on basic ajax in rails.

Colin