Write property tests¶

Basic usage¶

To evaluate a property we must invoke the function forAll like this:

@Test
fun isCommutative() = forAll { x: Int, y: Int ->
    x + y == y + x
}

forAll Will generate random inputs and evaluate the content of the lambda 200 times. If the lambda return false, it will immediately throws an AssertionError making the test fail.

So the test pass only if the lambda returns true for 200 random inputs.

Note

Kwik can automatically generate values for Int, Double, Boolean and String.

For other types we have to Create a custom generator

Use assertions¶

If writing a lambda that return a boolean is not of your taste, you may alternatively use checkForAll. Instead of returning a boolean, we have to throw an exception in case of falsification.

Example:

@Test
fun isCommutative2() = checkForAll { x: Int, y: Int ->
    assertEquals(x + y, y + x)
}

This alternative can be especially useful to get more descriptive messages. In the example above, a falsification of the property would display the expected and actual values. Theses kind of messages cannot be provided when using forAll.

Choose the number of iterations¶

By default the property is evaluated 200 times [1]. But we can configure it by setting the argument iteration.

For instance, the following property will be evaluated 1000 times:

forAll(iterations = 1000) { x: Int, y: Int, z: Int ->
    (x + y) + z == x + (y + z)
}

[1]	The default number of iterations can be configured via system property

Use a seed to get reproducible results¶

Because Kwik use random values, it is by definition non-deterministic. But sometimes we do want some determinism. Let’s say, for instance we observed a failure on the CI server once, how can be sure to reproduce it locally?

To solve this problem, Kwik use seeds. By default a random seed is used and printed in the console. If we observe a failure in the CI, we simply look at the build-log to see what seed has been used, then we can pass the seed to forAll so that it always test the same inputs.

forAll(seed = -4567) { x: Int ->
    x + 0 == x
}

Note

The seed can be set globally

Customize generated values¶

Random input is good. But sometimes, we need to constraint the range of possible inputs.

That’s why the function forAll accepts generators, and all built-in generators can be configured.

forAll(Generator.ints(min = 0), Generator.ints(max = -1)) { x, y ->
    x + y < x
}

Create a custom generator¶

But what if we want to test with input types which are not supported by Kwik, like domain-specific ones?

For this we can create a generator by implementing the interface Generator. And since that interface is a Kotlin fun interface, (aka SAM) one can create a custom generator like this:

val customGenerator1 = Generator { rng ->
    CustomClass(rng.nextInt(), rng.nextInt())
}

For enums or finite set of values we can use Generator.enum() and Generator.of():

val enumGenerator = Generator.enum<MyEnum>()

val finiteValueGenerator = Generator.of("a", "b", "c")

Note

You may reuse existing operators to build new ones. This can be done by calling Genarator.genarate(Random) on other operators, or by using the available operators

Add samples¶

Testing against random values is great. But often some values have more interest to be tested than others.

These edge-cases can be added to a generator with the function withSamples.

val generator = Generator.ints().withSamples(13, 42)

// since ``null`` and ``NaN`` are common edge-case, there are dedicated ``withNull`` and ``withNaN`` operators.
val generatorWithNull = Generator.strings().withNull()
val generatorWithNaN = Generator.doubles().withNaN()

The samples have higher chance to be generated and will be tested more often.

Note

All built-in generators already have some samples included.

For instance Generator.ints() will generate 0, 1, -1, Int.MAX_VALUE and Int.MIN_VALUE often.

Skip an evaluation¶

Sometime we want to exclude some specific set of input. For that, we can call skipIf in the property evaluation block.

forAll { x: Int, y: Int ->
    skipIf(x == y)

    x != y
}

Be careful to not overuse it though as it may slow down the tests. Always prefer creating or configuring custom generators if you can.

Make sure that a condition is satisfied at least once¶

All theses random inputs are nice, but we may want to be sure that some conditions are met all the time.

For that, we can call ensureAtLeastOne. It will force the property evaluation run as many time as necessary, so that the given predicate gets true.

forAll { x: Int, y: Int ->

    // This forces the property to run as many times as necessary
    // so that we make sure to always test the case where x and y are both zero.
    ensureAtLeastOne { x == 0 && y == 0 }

    x * y == y * x
}

Be careful to not overuse it either as it may slow down the tests.