Proving breaking changes in .NET Core – Part 3

In my previous post on this topic, I showed how we could use FsCheck to generate random values for a pre-defined piece of code, and then once a failing test has been found, would be able to "shrink" the input set until it found the simplest, most minimal repro. In this post, I want to show how you can use FsCheck to implement what is known as model-based testing.

FsCheck actually has its own Model Based Testing API, but in this post I'm going to use something hand-rolled that's a little simpler for this example.

What is Model Based Testing?

Model Based Testing (MBT) is a variant on Property Based Testing (PBT). With "normal" PBTs, you create a specific function (or set of functions) under test, and then use the PBT framework to generate random inputs for them. With MBTs, the framework will also generate the set of function calls as well as the arguments for them! A good example of this might be a data access layer: Let's say you test your Create, Update and Delete functions for your Customer table, each in isolation and they work fine. However a specific bug only shows itself if you create 3 customers, delete 2 and then update the last one. Such a test might be very difficult to identify up-front, and this is where MBTs come in.

Model Based Testing and the LINQ bug

In the previous post, I only tested pre-defined methods / chains e.g. OrderBy().FirstOrDefault() etc. What we're going to use FsCheck to do is to create random chains of LINQ methods for us, using both .NET Framework and .NET Core, and see what happens.

Start with Types

As usual in F#, we start by using types to define our domain. In this case, a record that explains what FsCheck needs to generate for us:

type GeneratedInputs =
    {
        Data : int NonEmptyArray // generated input data
        Mappers : Mapper list    // generated list of mappers
        Aggregator: Aggregator // generated aggregator
    }

Mappers and Aggregators are themselves two types, both discriminated unions:

type PredicateFunction = Function<int, bool>
type IdentityFunction = Function<int, int> // Not really "identity" but something that returns a value of the same type as input

type Aggregator =
    | FirstOrDefault of PredicateFunction
    | LastOrDefault of PredicateFunction
    | SingleOrDefault of PredicateFunction
    | Count of PredicateFunction

type Mapper =
    | Select of IdentityFunction
    | Where of PredicateFunction
    | OrderBy of IdentityFunction
    | SelectMany of Function<int, int array>
    | Take of int

Generate data and functions

Here, we define the kind of Aggregations and Mappings that we want FsCheck to generate, along with the actual implementations for the function arguments that they take. For example, Where takes in an argument that is a predicate - FsCheck will happily generate a predicate function of int -> bool for us. So it could generate us a predicate designed for LastOrDefault that returned false for the value 10, but true the rest of the time; similarly, for Mapper functions, it'll generate the appropriate functions for us. So a sample generated GeneratedInputs value might look like this:

{
    Data = NonEmptyArray [|0; 2; 1|]
    Mappers = [OrderBy { 10->-1 }; Select { 10->0 }; Where { 10->true }]
    Aggregator = LastOrDefault { 10->false }
}

Here, FsCheck has generated sample input of an array of 0, 2 and 1, and wants to run it on a chain of three mappers: OrderBy, Select and the Where, followed by an aggregation function call to LastOrDefault. Once we have this chain, we can essentially iterate over every mapper, calling both the .NET Framework and .NET Core implementations and ensure that the end result is the same (again, as in the previous blog post, this is both the final value as well as the number of calls to each function argument). Here's an example of it finding the same OrderBy -> FirstOrDefault issue, but by using model based testing:

Reproducing the issue

Failed after 55 tests. Parameters:

{ Data = NonEmptyArray [|2; 0; -1|]
  Mappers =
           [OrderBy { -1->1; 0->1; 1->0; 2->1 };
            OrderBy { -1->0; 0->0; 1->0; 2->0 }]
  Aggregator = FirstOrDefault { -1->false; 0->false; 1->false; 2->true }}

Shrunk 3 times to:
{ Data = NonEmptyArray [|2; 0|]
  Mappers = [OrderBy { -1->0; 0->0; 1->0; 2->0 }]
  Aggregator = FirstOrDefault { -1->false; 0->false; 1->false; 2->true }}

expected: { Result = 2; Calls = 1 }
actual:   { Result = 2; Calls = 2 }

Again, let's go through this bit by bit:

1. FsCheck found a failure with an array of three numbers as a dataset, ordered it twice and then ran FirstOrDefault on it.
2. It expected the result to be 2, and only make one call to FirstOrDefault. However, the .NET Core implementation needed 2 calls.
3. It then shrunk the test case to just a single OrderBy followed by that call to FirstOrDefault.

The source code for this can be found here

Outcome

Interestingly, if you update the test project from netcore3.1 to net5, this specific issue has indeed been fixed and the optimisation reverted to reflect the original behaviour. However, the change to LastOrDefault is still in .NET5:

{ Data = NonEmptyArray [|-4; 0; 0|]
  Mappers = []
  Aggregator = LastOrDefault { -4->false; 0->true } }

Shrunk 1 times to:
{ Data = NonEmptyArray [|0; 0|]
  Mappers = []
  Aggregator = LastOrDefault { -4->false; 0->true } }

expected: { Result = 0; Calls = 2 }
actual: { Result = 0; Calls = 1 }

This is because LastOrDefault in .NET Core counts from the back of the list rather than from the front, and so can stop on the first match, whereas the original implementation had to traverse the entire list to find the final match.

Summary

FsCheck is often used for generating inputs to static, predefined function calls. But you can also use it to generate entire function chains - or commands, or events etc.. This is extremely useful to ensure that your functions integrate correctly with one another and can often find issues that you won't easily consider yourself.