How good are your .NET tests? Test your tests with Stryker mutator

Imagine you’re starting to develop a new feature… and this time you decide to do it right. So, you started with Test-Driven development, because it’s part of your ethic to write tests (a surgeon never asks whether they should wash their hands before an surgery). It felt so good. And it felt so right. You’re satisfied with your code design and the fact that you’ve started with tests. And hey, the code coverage is pretty good too!

You deployed this code to production and it is working really well. Feels pretty good with those green tests, huh?

Not so long after…

The feature is cool and the customer tells you that you’ve done really good job. They have an idea to make this feature even better.

You’re a proactive developer, so you start digging a little bit, trying to understand what problem the customer has and how to solve it in the best way. After a short discussion, you know what to do – time to revisit your code and extend it a little bit!

Who wrote this shi…stuff?!

No, I am just kidding. You just understood the problem better (learning the business domain is a very important part of our job), and you have an idea of how to tackle this problem in a better way. And it’ll be easier for you to introduce this new feature if you refactor this code. Just a little bit.

It turned out that refactoring code was a really good idea. With the new structure, it was really simple to introduce what the customer needs.

New functionality got new tests. The old ones are still passing (perhaps a few changes were introduced to the test code)

What do you mean by stopped working?

Suddenly, you get a message from customer (not so happy anymore…).

This song probably fits your reaction when getting such messages.

Turns out, the previously delivered part of the new feature stopped working. The new is fine… but the bug was introduced to the previous part.

How did it happen? The tests were green. Coverage was high. You checked the git log, no one modified this code…

Turns out…

Turns out that you haven’t covered all the cases with your tests, despite 100% coverage. Then you changed few things here and there, and in the effect, the functionality doesn’t work as it should. How to reduce the chance of such a situation occurring again?

Stryker Mutator To The Rescue

Before I tell you what is Stryker, let’s talk about mutation testing.

What is mutation testing?

Mutation testing is all about introducing bugs (mutants) into your production code. The tests are run for each mutant. The tests should fail. Failing means that the mutant is killed. If the tests pass – the mutant survived, which means that there are false positives results.

So, Stryker does all of it automatically for you. Stryker supports different mutators, for example, arithmetic, equality, logical operators, or even LINQ. You can check the full list of available mutators in documentation https://stryker-mutator.io/docs/stryker-net/Mutators

Show Me The Code

Repository

For this exercise I used https://github.com/dotnet-architecture/eShopOnContainers. It contains well a written domain code and unit tests. We’ll focus on the Order Aggregate example (class, tests).

Running Mutant

Running Stryker mutator is simple. Most of the times you just need to run the dotnet stryker command in the CLI. When the test project contains more than one reference to production code, you have to choose a specific project file. Stryker is very helpful there and it provides an example path. To make the result more readable for purpose of this blog post, I used mutated only the file linked above.

First mutant test results

Stryker supports multiple reporters. For the purpose of this blog post I used dashboard reporter.

OrderAggregate Mutation Tests
OrderAggregate Mutation Tests

Order’s mutation score is 11.54%. 26 mutants very introduced (mutations of the production code) and only 3 were killed. Let’s dig a little bit deeper and see what has been mutated in the Order class.

SetAwaitingValidationStatus mutation
SetAwaitingValidationStatus mutation

We can see that Stryker used Equality mutation in this case. It simply changed “==” to “!=”. The result tells us that there’s no test that would detect such change.

Introduce the test

In OrderAggregateTests.cs I added following test

   [Fact]
   public void      Order_status_can_be_changed_to_awaiting_validation_only_for_submitted_order()
   {
        var address = new AddressBuilder().Build();
        var order = new OrderBuilder(address)
            .AddOne(1, "cup", 10.0m, 0, string.Empty)
            .AddOne(1, "cup", 10.0m, 0, string.Empty)
            .Build();

        order.SetCancelledStatus();
        order.SetAwaitingValidationStatus();

        var domainEvents = order.DomainEvents;

        Assert.Equal(domainEvents.Count, 2);

        foreach(var @event in domainEvents) 
        {
            Assert.False(@event.GetType() ==        typeof(OrderStatusChangedToAwaitingValidationDomainEvent));
        }
   }

This test:

  • Creates new order
  • Sets status to Cancelled
  • Sets status to AwaitingValidationStatus
  • and then it checks whether only 2 domain events were created
  • but most importantly it checks whether OrderStatus was not changed to AwaitingValidation

Mutant test run after introducing the test

Mutant test run after introducing test
Mutant test run after introducing test

The test killed more mutants than expected 😇. Because of introducing the test, results look a little bit better. So, if there would be a person refactoring the Order class and the person would mess up the logic around AwaitingValidationStatus, our tests would discover that fact.

SetAwaitingValidationStatus mutation test result after introducing the test
SetAwaitingValidationStatus mutation test result after introducing the test

Summary

Mutation testing is about testing the tests, to minimise the chance of getting false positives. Stryker is a great tool that helps you with Mutation Testing.

The question that you might ask yourself is – when should you run those mutant tests? In CI/CD? Locally? Once in a while?

My answer is… I don’t know, yet. I am still new to mutation testing. As for now, it makes the most sense to me to add it as a part of my development routine. That means when I start working with some code (let it be new or legacy one) I run Stryker to see whether I have good mutant coverage here. Then I have more confidence to work with that code. If I don’t – then I focus on increasing that coverage first, before further development. If you’d prefer to have it as a part of your CI/CD, it is possible. Take a look at the documentation https://stryker-mutator.io/docs/stryker-net/Stryker-in-pipeline

Update:

I’ve added second part of this post, explaining First <-> FirstOrDefault LINQ mutations

Let's stay in touch!