Property Based Testing - RichardWarburton/lambda-behave GitHub Wiki

Whilst we usually write tests which focus on specific testcases a lot of the time those testcases are really only examples of a broader property that holds about your software. For example if you were testing a reverse method then you might check that if you reverse "abc" twice then you end up with "abc" again. Actually the overall property that you're testing is that if you reverse a String twice then you end with the same String again.

Property based testing is an approach to testing that focuses on describing the overall property without hard coding specific examples of it. You can think of it as like generating your testcase examples automatically. Lambda Behave isn't the first framework to implement this approach - the quick check and scala check projects are other examples of property based testing.

The API for this is similar to data driven specifications but instead of listing a table of values it gives for control over the way that the values are generated and how many need to be generated. Here is an example of how to show that reversing a String twice returns the same String using randomly generated test case values.

it.requires(10)
  .example(asciiStrings())
  .toShow("reversing a String twice returns the original String", (expect, str) -> {
      String same = new StringBuilder(str).reverse().reverse().toString();
      expect.that(same).isEqualTo(str);
  });

All generated specifications follow this common pattern where;

The require clause expresses how many values to generate, The example clause states what type of objects to generate and how to generate them, This is overloaded to allow multiple columns of testcase values to be generated. The toShow clause behaves like a toShow clause for a data drive spec. It is type safe against the the different columns. So in the above example the paramter str will have had its type correctly inferred as String.

One of the things to bear in mind with using random testcase generation in order to check properties is that it can make it hard to isolate test failures. Suppose you have a property that fails 1 in time in 1000 and you run your tests with 10 examples and it fails. You don't necessarily want to have to re-run it repeatedly until you're sure that it passes again. You want the ability to deterministically re-run tests that have failed.

Lambda Behave supports that use case by printing out the seed that was used to run the testcase generation. This allows you to set the seed in future runs and ensure deterministic testcase generation. If you look at the outputted logs when you run a test (from your IDE or build system) you will see an entry that looks like: (seed: 119003056321043) for a test run. Here's an example of the output:

0: reversing a String twice returns the original String(AO9=4|) (seed: 119003056321043) has succeeded
1: reversing a String twice returns the original String(Px4OP=) (seed: 119003056321043) has succeeded
2: reversing a String twice returns the original String(+pFJ{5!2o) (seed: 119003056321043) has succeeded

You can now set this seed in your random testcase by adding a SourceGenerator. This is the generator whatever the source of random numbers that's used in the random testcase generation. The builtin randomNumbers generator has a factory method that takes a seed as its only argument. So we can refactor the above String reversal test to use a deterministic seed as follow:

import static com.insightfullogic.lambdabehave.generators.SourceGenerator.randomNumbers;

...

it.requires(10)
  .withSource(randomNumbers(101L))
  .example(asciiStrings())
  .toShow("reversing a String twice returns the original String with deterministic seed", (expect, str) -> {
      String same = new StringBuilder(str).reverse().reverse().toString();
      expect.that(same).isEqualTo(str);
  });