Proving breaking changes in .NET Core – Part 2

In my previous post on this subject (several months ago now, I admit!), I provided a high level overview of what the potential breaking change with .NET Core's LINQ was, and how you might build a function to compare an old and new implementations of another function. The function I built compared the two implementations by result and interactions with higher order functions.

Armed with this, we could then take the original .NET Framework implementation of LINQ and compare it with the .NET Core version, to prove what the potential breaking changes were.

Using FsCheck to test hard-coded method combinations

One surprise was that FsCheck immediately highlighted a difference in behaviour between the .NET Framework and the .NET Core 3 implementations of LastOrDefault (not a method that was specified in the GitHub issue):

testProperty "LastOrDefault has not changed" <| LinqCompare.Check (Enumerable.LastOrDefault, OldEnumerable.LastOrDefault)

Notice that all we provide are the two "versions" of the LINQ method. FsCheck will then run each test 100 times, creating not only a randomly distributed set of input data (starting with a small array and gradually increasing in size and complexity) but also random functions that are passed as the predicate. That's right - FsCheck will even generate code for us. And sure enough if we test this out, we can see that LastOrDefault has in fact changed between Framework and Core:

[ERR] All Tests.Hard Coded.Comparing Core and Framework.Single method.LastOrDefault has not changed failed in 00:00:00.3540000.
Failed after 10 tests. Parameters:
        [|-1; 0|] { -1->false; 0->true }
Shrunk 2 times to:
        [|0; 0|] { 0->true }
Result:
        Exception
  Expecto.AssertException: Both results should be the same.
expected:
Ok { CallCount = 2
     Result = 0 }
  actual:
Ok { CallCount = 1
     Result = 0 }

Let's decompose this message, one bit at a time:

• FsCheck ran ten tests comparing the original and new LastOrDefault implementations.
• It stopped after ten tests because it found some input data that broke our test that checks that the result and behaviour of old and new versions must be the same: [| -1; 0 |] and a predicate that returns false for -1 and true for 0.
• It then simplified the test inputs (a process known as "shrinking") to the smallest possible inputs which still break, [| 0; 0 |] and (0 -> true).
• It provided the two different outputs; although the result of both predicate calls was the same, the original implementation made two calls to the supplied predicate whereas the newer version only made one.

If you're interested, you can check out the old LastOrDefault implementation and the new LastOrDefault implementation. As you can see, the new .NET Core implementation counts backwards from the end of the list, whereas the .NET Framework implementation starts at the beginning and counts forward, which explains the different behaviour above.

Let's now prove the issue raised in the original GitHub issue that I linked to in the previous post, which is that running a query on a collection with OrderBy followed by FirstOrDefault behaves differently.

• In .NET Framework, the predicate argument in FirstOrDefault will be called at most once.
• In .NET Core, the the predicate argument in FirstOrDefault will be called for every item in the collection.

Here's the test code:

testProperty "FirstOrDefault has not changed Ex" <|
    LinqCompare.Check (
        (fun (data, func) -> data.OrderBy(id).FirstOrDefault(func)), // .NET Core implementation
        (fun (data, func) -> OldEnumerable.FirstOrDefault(OldEnumerable.OrderBy(data, id), func)) // .NET Framework implementation
    )

Note that we use the id function, which is essentially a "no-op" function, in order to create an IOrderedEnumerable which is needed in order to replicate the issue. Also note that "old" implementation does not use extension methods but standard method calls.

And here's the result:

[ERR] All Tests.Hard Coded.Comparing Core and Framework.Run on OrderedEnumerable.FirstOrDefault has not changed failed in 00:00:00.4670000.
Failed after 13 tests. Parameters:
        [|0; 0; -1; 0|] { -1->false; 0->true }
Shrunk 2 times to:
        [|0; 0|] { 0->true }
Result:
        Exception
  Expecto.AssertException: Both results should be the same.
expected:
Ok { CallCount = 1
     Result = 0 }
  actual:
Ok { CallCount = 2
     Result = 0 }

With a collection of two elements (0, 0) and the FirstOrDefault predicate returning true for 0:

• .NET Framework: The predicate is only called once - for the first element.
• .NET Core: The predicate is called twice - for both elements in the collection.

This sample above proves that even though we find a match on the first element in the input collection, .NET Core 3 still calls the predicate twice!

Summary

In this post I showed how we can use F# and FsCheck to test for breaking changes between new and old versions of LINQ methods, and showed how to understand the output of FsCheck messages to see what's happening. Lastly, I showed how FsCheck shows the minimal failing test case for the specific GitHub issue in question.

In the next post in this series, I'll illustrate how we can use FsCheck to construct actual LINQ pipelines themselves, rather than just hard-coding them by hand.

In the meantime, you can check out the source code for this blog post. Until next time, have (fun _ -> :).

Isaac