One assertion per test, please

When it comes to unit or integration tests, there are of course good ones and bad ones. There are few things more treacherous than a test suite filled with low-quality tests that don’t test what they’re supposed to, if anything at all. When the test suite goes green, you’re not really any wiser.

To determine whether a test is good or bad, you can subject it to a number of rules. In this post, I will focus on one of the simpler:

There should only be one assertion per test!

I’m pretty adamant about this one, and it’s one of the first rules I bring up whenever I talk about how to write good tests. On the surface, the rule looks pretty simple, but there are actually a couple of existential questions behind it, such as: “What is a test?” and “What is an assertion?”

I was reminded of the rule when writing some tests using the JavaScript unit testing framework QUnit. I’ll come back to why in a moment, along with answers to the existential questions posed above, but let’s first look at the reasoning behind the rule.

Why only a single assertion?

The single assertion requirement has to do with clarity and conciseness. Here are some good reasons that motivate the rule:

A test should only be able to fail for a single reason, namely that the thing it tests doesn’t behave as expected.

Thing in this context refers to a particular aspect of the subject under test. Having multiple assertions means that a test can fail for multiple reasons, which makes it harder to determine the cause of failure. Furthermore, a failure in an early assertion hides subsequent assertions, meaning that the failure count doesn’t adequately represent the state of the test suite. In other words, things are worse than they look! It’s actually quite annoying to uncover new failures as you fix old ones.

It should be possible to name a test in a precise and informative way, so that a failure is as self-descriptive as possible.

With multiple assertions, a test really tests many things. This is of course difficult to capture in the name of the test, which typically results in a non-descriptive name, or a name that only describes part of what the test does. How about “TestCRUD” or “TestGet?”

Multiple aspects of the outcome of some action should be tested separately, and the tests should be able to succeed or fail independent of each other.

Dependencies between tests are bad (that’s another rule), and multiple assertions are in fact equivalent to multiple tests locked in a dependency chain. Tests that test successive behavior (e.g., one that tests that a method returns an empty list rather than null, and one that tests that the list is unmodifiable) are of course weakly dependent on each other, but typically in a way that when some fail, others are in an error state. But that’s ok - disregard the errors and focus only on the failures.

A test should be concise and readable.

Multiple assertions make a test more involved, and that hardly makes it concise. A test with a single assertion is usually easy to read and comprehend, and that’s important!

You might be able to come up with more reasons than these, and I’d love to hear them! I’d also be interested in hearing about counter-arguments and cases where you think having multiple assertions per test is valid. I can think of one or two, but that’s a topic for another post. :-)

Tips for writing single-assertion tests

When you write tests using the Arrange-Act-Assert or Given-When-Then styles in an ordinary test framework like NUnit, and put all parts within a single test, it’s tempting to write multiple assertions rather than having multiple tests that repeat the Arrange-Act/Given-When parts. Adopting a BDDish Context-Specification style makes life much easier. An example:

[TestFixture]
public class WhenSomethingHappens {
    [SetUp] // or [TestFixtureSetUp]
    public void Context() {
        // Prepare data and do "Something"
    }

    [Test]
    public void ThenFoo() {
        // Assert the "Foo" outcome
    }

    [Test]
    public void ThenBar() {
        // Assert the "Bar" outcome
    }   
}

This style has the added bonus of forcing you to separate different aspects of the subject under test into different test classes. (Having one test class per subject under test has its merits, but can turn into a mess rather quickly.)

The existential questions

As I wrote initially, the rule leaves some questions unanswered. To really know what the rule means, we have to define what a test is and what an assertion is.

Based on what I wrote earlier about tests failing independent of each other, a test must be something that can fail or be in an error state by itself, without other tests being affected. Therefore, we can define test to mean “isolated test execution unit.” In a test framework like NUnit, JUnit or TestNG, “isolated test execution unit” corresponds to a test method:

[Test] // NUnit
public void TestThatCalculatorCanAddNumbers() {
    Assert.AreEqual(3, calculator.Add(1, 2));
}

It can also be a smaller unit, like for example a lambda statement in NSpec:

// NSpec
public void given_a_calculator() {
    it["adding two numbers should result in a correct sum"] =
        () => calculator.Add(1, 2).should_be(3);
}

Thus, in NSpec the test isn’t the method, but the lambda statement. To be precise, an isolated test execution unit of a test framework is the smallest unit that the test runner of that framework can wrap within a try-catch statement.

What about assertion then? The answer is that it depends on the assertion framework you’re using (which can be the test framework itself, or some other framework). I usually think of assertion as a publicly facing assertion, meaning a method in the API of the assertion framework. Under the hood, such an assertion may be composite. For example, an assertion in the FEST Assert framework for Java typically performs some initial sanity assertions, for example that the list whose length should be verified isn’t null.

I realize that the definition of assertion is a bit murky, but bottom line I only want to see a single assertion statement in a test. And yes, I do think that Assert.IsTrue(a == b && c == d && ...) counts as cheating! ;-)

What about QUnit

As stated, I wrote some QUnit tests to test JavaScript code. To my surprise, one of the examples at the old QUnit page contained multiple assertions within a single test. Uh-oh, a violation of my precious rule! However, as it turns out, an early assertion failure does not hide later assertions - they run just fine. So this example:

test("multiple assertions", function() {
    equal(1, "2", "Failed!");
    equal(1, "1", "Passed!");
});

…results in this output:

Based on the report we see that QUnit seems to equate assertion and test, although the terminology is a bit unclear given that the assertions are specified in a function passed to the test function. This is interesting! If an assertion is a test, then the rule becomes a tautology, and as such, it’s pretty pointless. Furthermore, assertions in QUnit are clearly not exception based, which could mean that isolated test execution unit has nothing to do with try-catch, but is rather something that is entirely dependent on the test framework. I was on the verge of reconsidering the rule altogether, when in a later test, I tested an element-finding method by verifying the ID of the found element, like this:

test("a rule violation", function() {
    equal(findElement().id, "id1", "Null reference!");
    equal(1, "1", "Passed!");
});

Initially, findElement() returned null for the case tested, and lo and behold! The test runner choked altogether, since there is a try-catch statement wrapping all assertions, making the entire test function the isolated test execution unit:

Note also that the report says “0 tests of 1 passed” - the second assertion is completely hidden. Sure, we can use the expect(...) function to state how many assertions we make, but that just feels clumsy in this particular context.

Conclusion

In my eyes, the QUnit adventure confirms that the rule is indeed valid, for all the reasons outlined earlier! QUnit is simply not compatible with my opinions of how to write good tests. On a final note, I ditched it for Jasmine!