Write property tests¶
Basic usage¶
To evaluate a property we must invoke the function forAll
like this:
@Test
fun isCommutative() = forAll { x: Int, y: Int ->
x + y == y + x
}
forAll
Will generate random inputs and evaluate the content of the lambda 200 times.
If the lambda return false, it will immediately throws an AssertionError
making the test fail.
So the test pass only if the lambda returns true for 200 random inputs.
Note
Kwik can automatically generate values for Int
, Double
, Boolean
and String
.
For other types we have to Create a custom generator
Use assertions¶
If writing a lambda that return a boolean is not of your taste, you may alternatively use checkForAll. Instead of returning a boolean, we have to throw an exception in case of falsification.
Example:
@Test
fun isCommutative2() = checkForAll { x: Int, y: Int ->
assertEquals(x + y, y + x)
}
This alternative can be especially useful to get more descriptive messages. In the example above, a falsification of the property would display the expected and actual values. Theses kind of messages cannot be provided when using forAll.
Choose the number of iterations¶
By default the property is evaluated 200 times [1]. But we can configure it by setting the argument iteration
.
For instance, the following property will be evaluated 1000 times:
forAll(iterations = 1000) { x: Int, y: Int, z: Int ->
(x + y) + z == x + (y + z)
}
[1] | The default number of iterations can be configured via system property |
Use a seed to get reproducible results¶
Because Kwik use random values, it is by definition non-deterministic. But sometimes we do want some determinism. Let’s say, for instance we observed a failure on the CI server once, how can be sure to reproduce it locally?
To solve this problem, Kwik use seeds. By default a random seed is used and printed in the console.
If we observe a failure in the CI, we simply look at the build-log to see what seed has been used,
then we can pass the seed to forAll
so that it always test the same inputs.
forAll(seed = -4567) { x: Int ->
x + 0 == x
}
Note
The seed can be set globally
Customize generated values¶
Random input is good. But sometimes, we need to constraint the range of possible inputs.
That’s why the function forAll
accepts generators, and all built-in generators can be configured.
forAll(Generator.ints(min = 0), Generator.ints(max = -1)) { x, y ->
x + y < x
}
Create a custom generator¶
But what if we want to test with input types which are not supported by Kwik, like domain-specific ones?
For this we can create a generator by implementing the interface Generator
.
And since that interface is a Kotlin fun interface
, (aka SAM) one can create a custom generator like this:
val customGenerator1 = Generator { rng ->
CustomClass(rng.nextInt(), rng.nextInt())
}
For enums or finite set of values we can use Generator.enum()
and Generator.of()
:
val enumGenerator = Generator.enum<MyEnum>()
val finiteValueGenerator = Generator.of("a", "b", "c")
Note
You may reuse existing operators to build new ones. This can be done by calling Genarator.genarate(Random)
on other
operators, or by using the available operators
Add samples¶
Testing against random values is great. But often some values have more interest to be tested than others.
These edge-cases can be added to a generator with the function withSamples
.
val generator = Generator.ints().withSamples(13, 42)
// since ``null`` and ``NaN`` are common edge-case, there are dedicated ``withNull`` and ``withNaN`` operators.
val generatorWithNull = Generator.strings().withNull()
val generatorWithNaN = Generator.doubles().withNaN()
The samples have higher chance to be generated and will be tested more often.
Note
All built-in generators already have some samples included.
For instance Generator.ints()
will generate 0
, 1
, -1
, Int.MAX_VALUE
and Int.MIN_VALUE
often.
Skip an evaluation¶
Sometime we want to exclude some specific set of input. For that, we can call skipIf
in the property evaluation block.
forAll { x: Int, y: Int ->
skipIf(x == y)
x != y
}
Be careful to not overuse it though as it may slow down the tests. Always prefer creating or configuring custom generators if you can.
Make sure that a condition is satisfied at least once¶
All theses random inputs are nice, but we may want to be sure that some conditions are met all the time.
For that, we can call ensureAtLeastOne
. It will force the property evaluation run as many time as necessary, so that
the given predicate gets true.
forAll { x: Int, y: Int ->
// This forces the property to run as many times as necessary
// so that we make sure to always test the case where x and y are both zero.
ensureAtLeastOne { x == 0 && y == 0 }
x * y == y * x
}
Be careful to not overuse it either as it may slow down the tests.