7 Fatal Unit Tests Mistakes To Avoid

7 Fatal Unit Test Mistakes To Avoid

Some time ago in the reporting project that my team and I are creating, we decided to introduce unit tests on different layers. This happened after having some bugs (like WHERE clause removed due to inefficient refactoring 😛).

And if you’ve worked with reporting projects, you know that, at least in the beginning, there’s not much business domain, right?

Anyway, we decided to write more unit tests to check if the basic data requirements are okay. For example, if the user that we’re displaying is still active in the system.

One of the layers that we started testing was so-called, application layer. This layer contains classes that act as a glue between the API layer and Entity Framework DbContext.

Another layer that we started unit testing was the API layer. We used a very similar approach as described here. Everything was great. Less stress when introducing new features. We got the tests. We’re safe, right? Indeed. But…

The annoying thing was that EVERY time the business requirement changed, we had to adjust tests that were failing. The even worse thing is that those tests were failing, but the production code was okay!
If you know how frustrating that is, keep reading 😉

Okay. So it felt wrong… The approach that we’re using. I’ve done some research and read blog posts and books. Below are my finding on 7 fatal unit tests mistakes to avoid.

What is the goal of unit testing?

Let’s start with the basics. What is the goal of unit testing? It’s often said that unit tests lead to better architecture design. I couldn’t disagree with that. But… Is this why you write unit tests? Probably not. Code reviews, pair programming, or looking for inspiration on Github will help you develop better code architecture as well. So it cannot be the main goal. Let’s agree on the fact that better architecture of code is just a nice side effect of unit testing.

The main goal of unit testing is to enable sustainable growth of the project. If you ever started a project from scratch, you know how nice it grows. Until… Until it gets harder to sustain this growth over time. The reason for that is, the more features and infrastructure we have, the harder it gets to build something on top of that without introducing a regression bugs. It’s even possible to get to the point where fixing one bug introduces two other bugs. Who has been there 🙋‍♂️?

Systems gets too complex and disorganized. Fixing one issue leads to another. It’s terrible feeling!

This is like a domino effect. The system gets more and more complex and disorganized. One mistake leads to another mistake. Luckily! We have tests. And a good unit test is something that can help us overcome that. So the other goal of unit tests is to protect the codebase from regression bugs that occur when working on a new feature or fixing another bug.

Unit tests written with those goals in mind will help you and your team grow the project, keeping it sustainable and scalable.

What is a regression bug?

Trying to introduce new feature gets the old one broken. That’s regression.

Shortly. A regression bug is a software bug that makes a feature stop working as intended after a certain event. Such an event can be changing code base, hosting, and so on. We’ll focus on changing existing code because that’s where unit tests can help us.

1. Avoid Testing A Unit (Class)

One of the worst mistakes I’ve done in my career, related to unit tests, was tests that were oriented on unit testing class. Because I thought that was the “unit” in unit tests. I was wrong. When thinking about the unit, think about a single unit of behavior. Tests should tell a story about the problem that the code is solving.

Testing a class (as a unit) makes the test harder to understand. It’s hard to understand what they verify when you take the whole system into consideration.

There’s one more important factor. Unit testing class might lead you to have a good safety net against regression bugs. This has to be admitted. But, such tests are not at all maintainable. Every refactoring that will happen around that class will break that test. That’s because those tests are focused on implementation details, not the outcome. This will lead to less trust in tests. You might not want to run them often. You might not want to run them at all.

2. Avoid Coupling Between Unit Tests

I think that’s the second-worst mistake I’ve made when writing unit tests. Coupling them too much with each other. That is an anti-pattern! What does it mean for a test to be coupled to another test? Before I answer that, let me tell you what I think about well-designed unit tests. Well designed unit tests:

  • Run in isolation from each other
  • Don’t share state in test classes (for example, using Service Under Test as a private field of test class)
  • Are independent from each other. Modification from one test should never break another test.

Coupled unit tests violate one of those statements. An example is extracting part of arranging code (in Arrange, Act, Assert approach). Let it be preparing test data. Everything seems to be okay until there’s a test that needs different data set to start with. Then crazy combinations start. If statements, modifying data on fly. The tests become unreadable, hard to maintain. They end up being removed, or even worse, just skipped.

Rather than using some kind of sophisticated or complex setups, develop private factory methods. Those methods will help you quickly prepare the Arrange part. Act and Assert parts will remain simple 🙂

3. Avoid Trying To Write The Perfect Unit Test

Unfortunately, you cannot write perfect unit tests. Don’t even try. Perfect unit test would be:

  • 100% resistant to refactoring
  • Protect against regressions
  • And give fast feedback
You cannot have it all. You have to choose two that are the most important for your system.

You cannot have it all. You can only pick two. This is a little bit aside from unit tests, but which one you pick will probably depend on your project and types of tests that you want to develop. For example:
If you select end-to-end tests, they’ll be more resistant to refactoring than unit tests, for example. They’ll also do a good job protecting against regression bugs. But they won’t be fast. I am always trying to aim in resistance to refactoring.

Why?

I think that it is important to have tests that are also checking critical business parts of the system after refactoring. Those are the tests that you will trust and respect their results.
Tests that force you to change assertion whenever you refactor part of production code will not gain your trust. This will lead to a high chance of you either stop writing unit tests or accepting that fact and writing unit tests just for sake of it.

4. Avoid Assertions For Multiple Behaviors

As mentioned earlier, focus on units of behaviors. This will let you develop tests that check the real business requirements. If you keep in mind to check the single unit of behavior, this will help you write good & valuable tests. It doesn’t mean it has to be just one assert per test!
You could have multiple asserts in one test, that are related to one story that the test is telling. Or the business requirement if you will.

5. Avoid Focusing On Hows Instead Of Whats

It’s very tempting to mock everything that you can and just assert for the result that you set up. Or to check that method X was called after method Y, N times.
Isn’t it? Have you done it before thinking that is the right thing to do? ✋ This kind of test is fast. That’s for sure.
But are they reliable?
Are they resistant to refactoring?
Are they able to point out regression bugs?
I highly doubt (because I did that and they didn’t help 🤦‍♂️).

The point is to focus on WHAT is done inside the system under test. Not HOW. Focusing on HOW the system under tests handles its stuff is leading to fragile tests that are not resistant to refactoring. The reason for that is focusing on technical implementation. The tricky part is that the implementation can be changed/enhanced, but the outcome of the business method will be the same!

I’ll repeat my self, but that will lead to tests that are not resistant to refactoring. And…
Resistance to refactoring should be non-negotiable.

  • Aim at gaining as much of it as possible, with keeping tests reasonably quick
  • Then choose either for faster tests vs tests that are good at catching regression bugs (look at point 3).

6. Avoid Trivial Unit Tests

Trivial unit tests are testing the logic that is unlikely to break. Trivial unit tests have some advantages, like:

  • They do provide fast feedback.
  • They do run quickly.
  • They’re resistant to refactoring
    But they will not reveal regression. That’s why I think you should avoid doing trivial tests.
public class Person
{
    public string FirstName { get; set; }
    /* Other person properties */
}

public class PersonTests
{
    public void returns_exactly_the_same_first_name_that_was_set() {
        // Arrange

        var person = new Person();

        // Act
        person.FirstName = "Joe";

        // Assert
        person.FirstName.Should().Be("Joe");
    }
}

7. Avoid Writing Tests For Coverage Metrics

Sometimes in the industry, you can hear that 100% of code coverage will lead you & your project to amazing things… It won’t. Unless you consider changing your test code every time you make a change in production code. I don’t think that is amazing. Also, no one is going to give you an achievement for having 100% coverage metrics. They won’t.

So why people are talking about 100% test coverages?

If you ask me, they do it for a reason. They know you won’t probably test 100% of your code. But you might be able to test 100% of your important code. For example, it’s always a better investment to focus on business domain related code, than on trivial things. What I keep in mind is:
Try to have a reasonably high level of code coverage in core parts of the system. But don’t let it be the top priority. Metrics might not be always reliable and can lead to bad project development.

What might happen?
Developers could start trying to test every single line of code. This leads us back to changing the line of tests code when changing part of code that doesn’t have a critical business effect.

SUMMARY

  • Good tests provide maximum value with minimum maintenance cost. Hence, the unit tests should test important parts of the codebase and do it well.
  • There’s no point in having tests that are not run in CI/CD.
  • No one is going to reward you for having a lot of tests. Neither for high code coverage.
  • Keep in mind that tests are also the code that you have to maintain.
  • Focus on WHATS (the outcome) not the HOWS (technical implementation details)
  • Use Mocking frameworks wisely
  • Resistance to refactoring is non-negotiable.
  • Don’t try to write a perfect unit test. Pick two out of three attributes and max them out with permission to have less priority on the last attribute.
  • When thinking about unit tests, think about the story that they’re trying to tell. Don’t focus on the testing unit of class. Focus on testing unit of behavior.
  • And once again, resistance to refactoring