With pytest, what's the benefit of using parameters instead of multiple assert statements for a test?
Question:
I’m reading over documentation and blogs about PyTest, but am not finding a good explanation for why you’d use @pytest.mark.parametrize instead of multiple assert statements when running a test. Why use one versus the other? What’s the benefit of using mark.parametrize, if there is one?
I came across a comment for another testing framework that said something about parametrizing continuing to run all parameters instead of stopping at the first failure, but I tested this and it didn’t seem to be true for PyTest:
def summing(x, y):
return x + y
def test_summing():
assert summing(1, 2) == 3
assert summing(3, 4) == 8
assert summing(4, 5) == 9
@pytest.mark.parametrize('x, y, result', [(1, 2, 3), (3, 4, 8), (4, 5, 9)])
def test_summing_param(x, y, result):
assert summing(x, y) == result
Both of the tests stopped after the second test (which failed)
$ pytest param_test.py
========================= test session starts ==========================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/coni/src/Scraping
collected 4 items
param_test.py F.F. [100%]
=============================== FAILURES ===============================
_____________________________ test_summing _____________________________
def test_summing():
assert summing(1, 2) == 3
> assert summing(3, 4) == 8
E assert 7 == 8
E + where 7 = summing(3, 4)
param_test.py:8: AssertionError
______________________ test_summing_param[3-4-8] _______________________
x = 3, y = 4, result = 8
@pytest.mark.parametrize('x, y, result', [(1, 2, 3), (3, 4, 8), (4, 5, 9)])
def test_summing_param(x, y, result):
> assert summing(x, y) == result
E assert 7 == 8
E + where 7 = summing(3, 4)
param_test.py:13: AssertionError
======================= short test summary info ========================
FAILED param_test.py::test_summing - assert 7 == 8
FAILED param_test.py::test_summing_param[3-4-8] - assert 7 == 8
===================== 2 failed, 2 passed in 0.07s ======================
Answers:
I think that the biggest advantage of using parametrize() is that it avoids duplicating test code. This duplication isn’t a big problem in your example, because each test is only one line. But most tests aren’t that simple. For example, if you are testing a function with complex inputs, it might take a few lines to setup those inputs. Or if you are testing a function with complex outputs, you might want to assert several things about those outputs. If you don’t use parametrize() in cases like these, you end up writing the same code over and over again for each set of inputs and outputs, and there are a bunch of downsides to that: it gets hard to read the tests, hard to refactor the tests, easy to introduce subtle differences between test cases, etc.
In addition to the above, there are a few ancillary benefits to using parameters that are worth mentioning:
- Sometimes it’s useful to reuse parameters between tests.
- They make it easier to load tests from an external source, e.g. a config file. Sometimes test cases can be specified much more succinctly in YAML than in python.
- They lower the barrier to adding more tests, which I find leads to having a more complete test suite.
- They give better output about the inputs that led to a failed test.
A test function consists of these steps:
- arrange
- act
- assert
We use multiple assert statements when we have the same arrange and act in several tests.
We use @pytest.mark.parametrize() when we have the same act and assert in several tests.
These could help us avoid writing duplicate code and enhance readability and maintainability of test code.
In this example, parametrizing tests does the trick because we have several arranges.
I’m reading over documentation and blogs about PyTest, but am not finding a good explanation for why you’d use @pytest.mark.parametrize instead of multiple assert statements when running a test. Why use one versus the other? What’s the benefit of using mark.parametrize, if there is one?
I came across a comment for another testing framework that said something about parametrizing continuing to run all parameters instead of stopping at the first failure, but I tested this and it didn’t seem to be true for PyTest:
def summing(x, y):
return x + y
def test_summing():
assert summing(1, 2) == 3
assert summing(3, 4) == 8
assert summing(4, 5) == 9
@pytest.mark.parametrize('x, y, result', [(1, 2, 3), (3, 4, 8), (4, 5, 9)])
def test_summing_param(x, y, result):
assert summing(x, y) == result
Both of the tests stopped after the second test (which failed)
$ pytest param_test.py
========================= test session starts ==========================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/coni/src/Scraping
collected 4 items
param_test.py F.F. [100%]
=============================== FAILURES ===============================
_____________________________ test_summing _____________________________
def test_summing():
assert summing(1, 2) == 3
> assert summing(3, 4) == 8
E assert 7 == 8
E + where 7 = summing(3, 4)
param_test.py:8: AssertionError
______________________ test_summing_param[3-4-8] _______________________
x = 3, y = 4, result = 8
@pytest.mark.parametrize('x, y, result', [(1, 2, 3), (3, 4, 8), (4, 5, 9)])
def test_summing_param(x, y, result):
> assert summing(x, y) == result
E assert 7 == 8
E + where 7 = summing(3, 4)
param_test.py:13: AssertionError
======================= short test summary info ========================
FAILED param_test.py::test_summing - assert 7 == 8
FAILED param_test.py::test_summing_param[3-4-8] - assert 7 == 8
===================== 2 failed, 2 passed in 0.07s ======================
I think that the biggest advantage of using parametrize() is that it avoids duplicating test code. This duplication isn’t a big problem in your example, because each test is only one line. But most tests aren’t that simple. For example, if you are testing a function with complex inputs, it might take a few lines to setup those inputs. Or if you are testing a function with complex outputs, you might want to assert several things about those outputs. If you don’t use parametrize() in cases like these, you end up writing the same code over and over again for each set of inputs and outputs, and there are a bunch of downsides to that: it gets hard to read the tests, hard to refactor the tests, easy to introduce subtle differences between test cases, etc.
In addition to the above, there are a few ancillary benefits to using parameters that are worth mentioning:
- Sometimes it’s useful to reuse parameters between tests.
- They make it easier to load tests from an external source, e.g. a config file. Sometimes test cases can be specified much more succinctly in YAML than in python.
- They lower the barrier to adding more tests, which I find leads to having a more complete test suite.
- They give better output about the inputs that led to a failed test.
A test function consists of these steps:
- arrange
- act
- assert
We use multiple assert statements when we have the same arrange and act in several tests.
We use @pytest.mark.parametrize() when we have the same act and assert in several tests.
These could help us avoid writing duplicate code and enhance readability and maintainability of test code.
In this example, parametrizing tests does the trick because we have several arranges.