PropCheck: property-based testing for your Ruby and Rails projects!

[Qqwy (Qqwy / Wiebe-Marten)]

Hi everyone!

Property-based testing is amazing: You specify what types of input values you expect and what kinds of properties are expected to hold true for all of those possible inputs, and the runtime is then able to automatically create and run thousands of test-cases for you. This often uncovers edge-cases that you might not have thought of if you were trying to come up with unit-test examples by hand.

What is even more cool is that when a test failure is encountered, the input can be shrunken back to the most simplest form that still results in an error, making the feedback very human-friendly.

Property-based testing can be considered a ‘pragmatic’ approach to ‘theorem-proving’. Mathematically proving that something holds for all cases is often very difficult, whilst stating that something should hold for all cases and then lobbing thousands of possible input sets at it is a lot easier to make. Of course this only allows to prove the presence (rather than the absence) of bugs, but because of the large number of input sets (and the fact that every time you run the test, yet more different input sets are tried), it converges to the same.

Besides working in a couple of language where property-testing is wide-spread I do quite a bit of consulting work in Ruby (usually in combination with Rails). Until now, Ruby did not have a mature library to do property-based testing. So I wrote one !

PropCheck

It features:

• Generators for common datatypes.
• An easy DSL to define your own generators (by combining existing ones, or make completely custom ones).
• Shrinking to a minimal counter-example on failure.

Usage Example

Here we check if naive_average indeed always returns an integer for any and all arrays of integers we can pass it:

# Somewhere you have this function definition:
def naive_average(array)
  array.sum / array.length
end

# And then in a test case:
include PropCheck::Generators
PropCheck.forall(numbers: array(integer)) do |numbers:|
  result = naive_average(numbers)
  unless result.is_a?(Integer) do
    raise "Expected the average to always return an integer!"
  end
end

When running this particular example PropCheck very quickly finds out that we have made a programming mistake:

ZeroDivisionError: 
(after 6 successful property test runs)
Failed on: 
`{
    :numbers => []
}`

Exception message:
---
divided by 0
---

(shrinking impossible)
---

Clearly we forgot to handle the case of an empty array being passed to the function. This is a good example of the kind of conceptual bugs that PropCheck (and property-based testing in general) are able to check for.

(If we were e.g. using RSpec, we might have structured the test as follows:

describe "#naive_average" do
  include PropCheck
  include PropCheck::Generators

  it "returns an integer for any input" do
    forall(numbers: array(integer)) do |numbers:|
      result = naive_average(numbers)      
      expect(result).to be_a(Integer)
    end
  end
end

)

PropCheck comes with many built-in data-generators and it is easy to build your own on top of these.

More information in the README on GitHub

---

I’m eager to hear your feedback !

[Delon_Newman]

I’ve been doing property-based testing on my projects, and started (though not really completed) a gem for it that I use (https://github.com/delonnewman/gen-test), but I’ll definitely check this out especially since it seems more complete that what I’ve put together.

I’m curious, how do you go about generating ActiveRecord instances? Do you make generators for each class or have you found good methods that are more general (like introspecting the class)?

[Qqwy (Qqwy / Wiebe-Marten)]

@Delon_Newman

I’m curious, how do you go about generating ActiveRecord instances? Do you make generators for each class or have you found good methods that are more general (like introspecting the class)?

Currently I am indeed manually writing a generator for each model class, e.g.

include PropCheck::Generators
user_gen = 
  fixed_hash({
    name: string, 
    age: nonnegative_integer
  })
  .map {|fields| User.new(**fields) }

However, a possibility could indeed be to make use of e.g. ActiveRecord::Base#columns_hash to fill this in with sensible defaults. Then the user would only need to manually override the fields that diverge from these defaults (like generating only valid email addresses for an email :string column).

The API could then be something like model_gen(User) or model_gen(User, email: email_gen).

@Delon_Newman

but I’ll definitely check this out especially since it seems more complete that what I’ve put together.

Give it a go and let me know how you fare!

[Delon_Newman]

Will do. I’m currently doing the same. I’ve done some work to make use the contracts from the “contracts” gem as generators. I’m curious if you have any thoughts about that.

You can see my attempts here: https://github.com/delonnewman/contracts-gen. I’d love to be able to specify contracts for my code then automatically generate tests based on them (Something the Clojure community is working towards).

[Qqwy (Qqwy / Wiebe-Marten)]

@Delon_Newman

I’m curious if you have any thoughts about that.

I think it is a great approach! I am actually in the process of builtin a contracts-library in Elixir that allows to use the same type-specifications to create runtime contracts, prop-test generators and improved documentation at the same time. (TypeCheck)

The tl;dr is: I think it is very valuable to combine contracts with property-based testing.

[cheerfulstoic (Brian Underwood)]

Nice, I love property-based testing, though I haven’t used it very much. I often forget to try it out and I think it works better in some situations than others. I wonder if there should be some sort of general guide showing practical examples. I see that in your README you’re property testing sort and average methods, but people rarely implement those or need to test them. I think it would help people who don’t already understand the concept of property-based testing to get into it. That said, I don’t know that your README needs to have those practical examples (I think it would be better with at least one practical example, but showing examples for sort and average also is approachable in a different way since everybody knows what those methods do), but if there was a guide out there it could be linked to.

[Qqwy (Qqwy / Wiebe-Marten)]

This is a great idea! Once I have a bit more time I’ll write a longer guide detailing the four common cases of property-based tests (modeling, generalizing examples, testing invariants, testing symmetric properties) using the library for the code snippets .

[Qqwy (Qqwy / Wiebe-Marten)]

Version 0.10 has been released! Besides a couple of bugfixes, the new version adds before/after/around callbacks that you can use to add setup and/or teardown logic that is run before/after/around every time a check with newly generated data is run.

That is really useful for instance when we want to use tools like DatabaseCleaner to efficiently property-test the behaviour of database queries.

After all, one very simple way to write a property-check, is to see if a complicated SQL-query does the same thing as a simple (but not as efficient) Ruby implementation of the same logic.

[nessamurmur (Nessa Jane Marin)]

Great job with this! Thank you for your hard work. I had contributed just a little to https://github.com/justincampbell/generative and spent about ~4 hours at a hackathon prototyping actual generators for it https://github.com/nessamurmur/degenerate and we never got around to one of the most important parts: shrinking. Glad to see someone put in the effort to build something more mature. Thanks much! I can’t wait to use it.