Improve Python testing with parameterisation

Parameterisation is a technique which makes testing simpler, more concise and more effective. It does this by separating test logic from test data. Let's consider a test for a simple function, and how it can be improved by parameterisation.

Simple test

Let's test the following function:

# prime.py
import math


def is_prime(x):
    """
    is_prime returns True or False indicating whether x is prime or not.
    """
    if x <= 1:
        return False
    for i in range(2, int(math.sqrt(x))):
        if x % i == 0:
            return False
    return True

We can test this for a range of inputs with the following code:

# test_prime.py
import unittest

from prime import is_prime


class TestIsPrime(unittest.TestCase):

    def test_x_negative(self):
        self.assertEqual(is_prime(-1), False)

    def test_x_zero(self):
        self.assertEqual(is_prime(0), False)

    def test_x_one(self):
        self.assertEqual(is_prime(1), False)

    def test_x_two(self):
        self.assertEqual(is_prime(2), True)

    def test_x_three(self):
        self.assertEqual(is_prime(3), True)

    def test_x_ten(self):
        self.assertEqual(is_prime(10), False)

    def test_x_fifty_three(self):
        self.assertEqual(is_prime(53), True)


if __name__ == "__main__":
    unittest.main()

We run the tests with:

$ python test_prime.py
Ran 7 tests in 0.000s

OK

All our tests pass, but our test code is verbose. Although each test is basically the same, we need to add a new method for each one. There is a lot of repeated boilerplate code.

Parameterisation

We can reduce this boilerplate by parameterising the tests:

# test_prime.py
import unittest

from prime import is_prime


class TestIsPrime(unittest.TestCase):

    def test_is_prime(self):
        test_cases = [
            (-1, False),
            (0, False),
            (1, False),
            (2, True),
            (3, True),
            (10, False),
            (53, True),
        ]
        for x, output in test_cases:
            self.assertEqual(is_prime(x), output)


if __name__ == "__main__":
    unittest.main()

We've extracted the test data into the test_cases variable. The test logic is then run on each of the test cases in turn. This is an improvement, but still has some flaws. When we run the test, we get the following output:

$ python test_prime.py
Ran 1 test in 0.000s

OK

The test output says we're running one test, even though we still have seven test cases. Even worse, if one of our tests fails, we aren't given any information on which test failed:

$ python test_prime.py
F
======================================================================
FAIL: test_is_prime (__main__.TestIsPrime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_prime_parameterised.py", line 19, in test_is_prime
    self.assertEqual(is_prime(x), output)
AssertionError: True != False

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

Parameterisation with subTest

Python 3.4 introduced a way to solve this problem. We can use the unittest.TestCase.subTest context manager for explicit parameterisation:

import unittest

from prime import is_prime


class TestIsPrime(unittest.TestCase):

    def test_is_prime(self):
        test_cases = [
            (-1, False),
            (0, False),
            (1, False),
            (2, True),
            (3, True),
            (10, True),
            (53, True),
        ]
        for x, output in test_cases:
            with self.subTest(name=str(x)):
                self.assertEqual(is_prime(x), output)


if __name__ == "__main__":
    unittest.main()

If a test fails, unittest will print out the name of the failed test:

$ python test_prime.py

======================================================================
FAIL: test_is_prime (__main__.TestIsPrime) (name='10')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_prime.py", line 20, in test_is_prime
    self.assertEqual(is_prime(x), output)
AssertionError: False != True

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

The subTest feature was only added in Python 3.4, but has been backported to Python 2.7 onwards with the unittest2 package.

Parameterisation with pytest

The pytest framework also solves these problems for us, and is compatible with Python 2. We can install it with pip install pytest.

Pytest contains a feature which allows us to parameterise test cases:

# test_prime.py
import pytest

from prime import is_prime


@pytest.mark.parametrize("x,output", [
    (-1, False),
    (0, False),
    (1, False),
    (2, True),
    (3, True),
    (10, False),
    (53, True),
])
def test_is_prime(x, output):
    assert is_prime(x) == output

Parameterisation is implemented with the pytest.mark.parametrize decorator. This decorator takes two arguments. The first is a comma separated string of the names given to the test cases defined in the second argument. The second argument is a list of tuples. Each tuple contains the data needed for a test case.

The decorator makes seven separate calls to test_is_prime, supplying each of the test cases in turn. When we run the tests, we see that seven tests are run:

$ pytest test_prime_pytest.py
test_prime_pytest.py .......

7 passed in 0.02 seconds

Importantly, when a test fails, pytest gives us information about which test failed:

$ pytest test_prime.py
test_prime_pytest.py ...F...

FAILURES
test_is_prime[2-False]

x = 2, output = False

>       assert is_prime(x) == output
E       assert True == False
E        +  where True = is_prime(2)

test_prime_pytest.py:16: AssertionError
1 failed, 6 passed in 0.04 seconds

Conclusion

Parameterising tests is a powerful technique. By separating test logic from data, the focus is shifted away from boilerplate code and onto testing features. It becomes trivial to add new test cases.

Parameterised tests work best with pure functions. A pure function is a function which satisfies the following constraints:

A pure function's behaviour depends only on the arguments passed to it, so they can often be exhaustively tested with a single parameterised test.

Change log: