Understand First and FirstOrDefault LINQ Mutations in Stryker Mutator

This blog post is a reply to doubt under Reddit’s publication of the previous blog post, which is an introduction to the Stryker Mutator. The doubts were related to Stryker LINQ mutations, especially First -> FirstOrDefault mutation and vice versa.

First things first

First, I have to admit that in the beginning, I didn’t understand it either. Even when I was presenting the Stryker for the first time to my team, I thought it just doesn’t understand the context of those First or FirstOrDefault (or Single and SingleOrDefault) being called. By the context, I mean any checks ensuring that the collection contains desired element before calling one of those methods. Luckily, those were explained to me on Twitter by the Stryker community. And to be honest, it’s brilliant. I found out that for quite a long time I was misusing those (First, Single and modifications) methods.

Example

Let’s get to the point and take a look at oversimplified Blog example.

public class Blog
    {
        private string _title;
        private string _author;

        public List<Post> Posts { get; }

        public Blog(string title, string author)
        {
            Posts = new List<Post>();
            _author = author;
            _title = title;
        }

        public Guid CreateNewPost(string title, string content)
        {
            Post post = new Post(title, content);
            Posts.Add(post);

            return post.Id;
        }

        public void ChangePostTitle_First(Guid postId, string title)
        {
            var post = Posts.First(x => x.Id == postId);

            post.ChangeTitle(title);
            // Change title
        }

        public void ChangePostTitle_FirstOrDefault(Guid postId, string title)
        {
            var post = Posts.FirstOrDefault(x => x.Id == postId);

            post.ChangeTitle(title);
            // Change title
        }

        public void ChangePostTitle_FirstOrDefault_With_Check(Guid postId, string title)
        {
            if (Posts.Any(x => x.Id == postId))
            {
                var post = Posts.FirstOrDefault(x => x.Id == postId);

                post.ChangeTitle(title);
            }
            else
            {
                throw new Exception("Post doesn't exist!");
            }
        }

        public void ChangePostTitle_First_With_Check(Guid postId, string title)
        {
            if (Posts.Any(x => x.Id == postId))
            {
                var post = Posts.First(x => x.Id == postId);

                post.ChangeTitle(title);
            }
            else
            {
                throw new Exception("Post doesn't exist!");
            }
        }

        public void ChangePostTitle_Find_With_Check(Guid postId, string title)
        {
            if (Posts.Any(x => x.Id == postId))
            {
                var post = Posts.Find(x => x.Id == postId);

                post.ChangeTitle(title);
            }
            else
            {
                throw new Exception("Post doesn't exist!");
            }
        }
    }

The Blog class can only do two things. First, it can create a new post, which in the effect is added to the Posts list. Second, it can change the post’s title with 5 different methods. The reason for 5 different methods is to show the differences between First, FirstOrDefault, Find LINQ methods, and resulting mutation results.

The First() LINQ method

Let’s take a look at the method that is using First() LINQ method and write tests against it.

        public void ChangePostTitle_First(Guid postId, string title)
        {
            var post = Posts.First(x => x.Id == postId);

            post.ChangeTitle(title);
            // Change title
        }

Let’s start with happy path test and check whether we can change the title of existing blog post.

        [Fact]
        public void First__post_title_can_be_changed()
        {
            // Arrange
            var blog = new Blog("Mutant Testing", "Lukasz");

            var postId = blog.CreateNewPost("LINQ Operators - First method", "Positive test for First Method");

            // Act
            blog.ChangePostTitle_First(postId, "LINQ Operators - First method. Should you use it?");

            // Assert
            Assert.Equal(blog.Posts.First(x => x.Id == postId).Title,
            "LINQ Operators - First method. Should you use it?");
        }

Of course this test is passing. Let’s see what the Stryker Mutator will tell us.

All mutants have been tested, and your mutation score has been calculated
All files [0/1 (0.00 %)]
├── Class1.cs [0/1 (0.00 %)]
│   └── [Survived] Linq method mutation (First() to FirstOrDefault()) on line 31
│       ├── [-] Posts.First(x => x.Id == postId)
│       └── [+] Posts.FirstOrDefault(x => x.Id == postId)

As expected – the First method has been changed to FirstOrDefault and there’s no test covering that case. Let’s fix it.

        [Fact]
        public void First__Exception_is_thrown_when_title_is_changed_for_non_existing_post()
        {
            // Arrange

            var blog = new Blog("Mutant Testing - First", "Lukasz");

            void ChangePostTitle() =>
                blog.ChangePostTitle_First(Guid.NewGuid(),
                 "Exception is thrown when the element is not found in the collection");

            // Act & Assert

            Assert.Throws<InvalidOperationException>(ChangePostTitle);
        }

And let’s check the mutation score.

All mutants have been tested, and your mutation score has been calculated
All files [1/1 (100.00 %)]
├── Class1.cs [1/1 (100.00 %)]
│   └── [Killed] Linq method mutation (First() to FirstOrDefault()) on line 31
│       ├── [-] Posts.First(x => x.Id == postId)
│       └── [+] Posts.FirstOrDefault(x => x.Id == postId)
[18:00:16 INF] Time Elapsed 00:00:26.3268479
[18:00:16 INF] The final mutation score is 100.00 %

Okay, so what just happened? Well, as we already know, mutation testing is all about inserting bugs into our production code. LINQ mutations are one of the available mutators for C# language. And because of that, Stryker changes First to FirstOrDefault. The reason for that is the difference of how those methods behave when there’s no result of when the collection is empty.

FirstOrDefaultFirst
Element is foundReturns a first element or first element matching given conditionReturns a first element or first element matching given condition
Element is not foundReturns default implementationThrows InvalidOperationException
Collection is emptyThrows argument null exceptionThrows argument null exception
First vs FirstOrDefault

Quite simple to fix that, isn’t it? We just have to add a valuable test.

But.. You might not want the code to throw the default exception. Or you might want to throw no exception at all… Then of course this code would be a little bit different. Let’s refactor it.

Custom exception

The method and test using First() and custom exception could look more or less like this

        public void ChangePostTitle_First_With_Check(Guid postId, string title)
        {
            if (Posts.Any(x => x.Id == postId))
            {
                var post = Posts.First(x => x.Id == postId);

                post.ChangeTitle(title);
            }
            else
            {
                throw new Exception("Post doesn't exist!");
            }
        }
        [Fact]
        public void First_with_Check_post_title_can_be_changed()
        {
            // Arrange
            var blog = new Blog("Mutant Testing", "Lukasz");

            var postId = blog.CreateNewPost("LINQ Operators - First method", "Positive test for First Method");

            // Act
            blog.ChangePostTitle_First_With_Check(postId, "LINQ Operators - First method. Should you use it?");

            // Assert
            Assert.Equal(blog.Posts.First(x => x.Id == postId).Title,
            "LINQ Operators - First method. Should you use it?");
        }

        [Fact]
        public void First_With_Check__Exception_is_thrown_and_title_is_not_changed_for_non_existing_post()
        {
            // Arrange

            var blog = new Blog("Mutant Testing - First", "Lukasz");

            // Act

            void ChangePostTitle() =>
                blog.ChangePostTitle_First_With_Check(Guid.NewGuid(), "Title");

            // Act & Assert

            Assert.Throws<Exception>(ChangePostTitle);
        }

Let’s check Stryker as well.

── Class1.cs [2/3 (66.67 %)]
│   ├── [Killed] Linq method mutation (Any() to All()) on line 61
│   │   ├── [-] Posts.Any(x => x.Id == postId)
│   │   └── [+] Posts.All(x => x.Id == postId)
│   ├── [Killed] Negate expression on line 61
│   │   ├── [-] Posts.Any(x => x.Id == postId)
│   │   └── [+] !(Posts.Any(x => x.Id == postId))
│   └── [Survived] Linq method mutation (First() to FirstOrDefault()) on line 63
│       ├── [-] Posts.First(x => x.Id == postId)
│       └── [+] Posts.FirstOrDefault(x => x.Id == postId)

66.67%… Not bad, it’s not 100%. The First -> FirstOrDefault mutant survived our tests. But it makes no sense… I made sure that the element is in the collection, so the default exception will never happen.

What can I do?

Remember when I said that Stryker knows nothing about method’s context? Well… it doesn’t. So now you have two choices:

  • Ignore First -> FirstOrDefault mutation – you can do it easily by adding following parameter to the command --ignore-methods "['First']"
  • Change it to Find. Wait, what?

Find method is not mutated by Striker. Probably because it doesn’t contain an alternative (like First -> FirstOrDefault or Single->SingleOrDefault) method. If you think about this, it makes sense. Consider situation when you’re working with an unexperienced developer that mostly uses FirstOrDefault (because tutorial said so). Potentially, there’s a danger that he would switch First to FirstOrDefault when he’d get an InvalidOperationException from First method.

Or… maybe there will be a mutation for Find method later on.

Also, according to this and this, Find() might improve the performance a little bit.

Now you know why Stryker is pointing the First -> FirstOrDefault mutations and hopefully I made it clear why it does, so it’s your conscious decision which method you’ll use and whether you’ll ignore First/FirstOrDefault methods in mutant testing or not.

The FirstOrDefault() LINQ method

As we went through the First() method, this section will be shorter. Let’s get straight into the method and happy path scenario test.

        public void ChangePostTitle_FirstOrDefault(Guid postId, string title)
        {
            var post = Posts.FirstOrDefault(x => x.Id == postId);

            post.ChangeTitle(title);
            // Change title
        }
        [Fact]
        public void FirstOrDefault_post_title_can_be_changed()
        {
            // Arrange
            var blog = new Blog("Mutant Testing", "Lukasz");

            var postId = blog.CreateNewPost("LINQ Operators - First method", "Positive test for First Method");

            // Act
            blog.ChangePostTitle_FirstOrDefault(postId, "LINQ Operators - FirstOrDefault method?");

            // Assert
            Assert.Equal(blog.Posts.First(x => x.Id == postId).Title,
            "LINQ Operators - FirstOrDefault method");
        }

And now, let’s run Stryker and see the result.

All mutants have been tested, and your mutation score has been calculated
All files [0/1 (0.00 %)]
├── Class1.cs [0/1 (0.00 %)]
│   └── [Survived] Linq method mutation (FirstOrDefault() to First()) on line 39
│       ├── [-] Posts.FirstOrDefault(x => x.Id == postId)
│       └── [+] Posts.First(x => x.Id == postId)

Similar story, we need to write a test that checks what will happen, if FirstOrDefault doesn’t find the Post with the given ID. Let’s do it and let’s check the mutation result!

        [Fact]
        public void FirstOrDefault__Exception_is_thrown_when_title_is_changed_for_non_existing_post()
        {
            // Arrange

            var blog = new Blog("Mutant Testing - First", "Lukasz");

            void ChangePostTitle() =>
                blog.ChangePostTitle_FirstOrDefault(Guid.NewGuid(),
                 "Exception is thrown when the element is not found in the collection");

            // Act & Assert

            Assert.Throws<NullReferenceException>(ChangePostTitle);
        }
All mutants have been tested, and your mutation score has been calculated
All files [1/1 (100.00 %)]
├── Class1.cs [1/1 (100.00 %)]
│   └── [Killed] Linq method mutation (FirstOrDefault() to First()) on line 39
│       ├── [-] Posts.FirstOrDefault(x => x.Id == postId)
│       └── [+] Posts.First(x => x.Id == postId)
[20:04:43 INF] Time Elapsed 00:00:25.7120064
[20:04:43 INF] The final mutation score is 100.00 %

So, the test checks whether the NullReferenceException is thrown if the Post was not found. That’s how it would behave, right? Adding that test, we kill the mutant.

But wait…

Normally, you probably would check whether you did get the Post object, right? So let’s refactor the code once again. In simplest way, we could do it like this:

        public void ChangePostTitle_FirstOrDefault(Guid postId, string title)
        {
            var post = Posts.FirstOrDefault(x => x.Id == postId);

            post?.ChangeTitle(title);
        }

Then the test, of course, fails. That’s because the exception is never thrown. But that’s not so good code. The caller of that method doesn’t know whether the title was changed or not. Now we can either throw a custom Exception or some kind of result. Let’s try with the result, followed by test and mutant coverage.

        public bool ChangePostTitle_FirstOrDefault(Guid postId, string title)
        {
            var post = Posts.FirstOrDefault(x => x.Id == postId);

            post?.ChangeTitle(title);

            return post != null;
        }
        [Fact]
        public void FirstOrDefault__False_result_is_returned_when_title_is_changed_for_non_existing_post()
        {
            // Arrange

            var blog = new Blog("Mutant Testing - First", "Lukasz");

            // Act 
            var result = blog.ChangePostTitle_FirstOrDefault(Guid.NewGuid(),
                 "Exception is thrown when the element is not found in the collection");

            // Assert

            Assert.False(result);
        }
All mutants have been tested, and your mutation score has been calculated
All files [1/1 (100.00 %)]
├── Class1.cs [1/1 (100.00 %)]
│   └── [Killed] Linq method mutation (FirstOrDefault() to First()) on line 47
│       ├── [-] Posts.FirstOrDefault(x => x.Id == postId)
│       └── [+] Posts.First(x => x.Id == postId)
[20:13:07 INF] Time Elapsed 00:00:24.2760898
[20:13:07 INF] The final mutation score is 100.00 %

Hopefully that answers how to take care of FirstOrDefault -> First mutation. Of course, if you’d check whether the item exists in collection, then once again, you’d get to the point where you’d have to choose between ignoring method’s mutation or using Find, instead. The code would look more or less like this:

        public void ChangePostTitle_FirstOrDefault_With_Check(Guid postId, string title)
        {
            if (Posts.Any(x => x.Id == postId))
            {
                var post = Posts.FirstOrDefault(x => x.Id == postId);

                post.ChangeTitle(title);
            }
        }

Looks similar, doesn’t it? 😅

To the point, Summary!

Hopefully now you understand what you can do about First and FirstOrDefault mutations (generally similar rules apply to Single and SingleOrDefault as well).

It’s often said and I totally agree with that sentence, we can code the same behaviour in multiple ways. Most of them have some pros and cons. There are rarely any silver bullets.

Stryker or Mutation Testing in general is not a silver bullet. I would rather call it a parachute or just safety net. Done coding your feature/bug? Make yourself and your team a favour and check whether your tests cover all possible cases. If they don’t, just fix it. If they do, congratulations, you rock ⭐️

Let's stay in touch!