How can I repeat each test multiple times in a py.test run?
Question:
I want to run each selected py.test item an arbitrary number of times, sequentially.
I don’t see any standard py.test mechanism for doing this.
I attempted to do this in the pytest_collection_modifyitems() hook. I modified the list of items passed in, to specify each item more than once. The first execution of a test item works as expected, but that seems to cause some problems for my code.
Further, I would prefer to have a unique test item object for each run, as I use id (item) as a key in various reporting code. Unfortunately, I can’t find any py.test code to duplicate a test item, copy.copy() doesn’t work, and copy.deepcopy() gets an exception.
Can anybody suggest a strategy for executing a test multiple times?
Answers:
One possible strategy is parameterizing the test in question, but not explicitly using the parameter.
For example:
@pytest.mark.parametrize('execution_number', range(5))
def run_multiple_times(execution_number):
assert True
The above test should run five times.
Check out the parametrization documentation: https://pytest.org/latest/parametrize.html
In order to run each test a number of times, we will programmatically parameterize each test as the tests are being generated.
First, let’s add the parser option (include the following in one of your conftest.py’s):
def pytest_addoption(parser):
parser.addoption('--repeat', action='store',
help='Number of times to repeat each test')
Now we add a “pytest_generate_tests” hook. Here is where the magic happens.
def pytest_generate_tests(metafunc):
if metafunc.config.option.repeat is not None:
count = int(metafunc.config.option.repeat)
# We're going to duplicate these tests by parametrizing them,
# which requires that each test has a fixture to accept the parameter.
# We can add a new fixture like so:
metafunc.fixturenames.append('tmp_ct')
# Now we parametrize. This is what happens when we do e.g.,
# @pytest.mark.parametrize('tmp_ct', range(count))
# def test_foo(): pass
metafunc.parametrize('tmp_ct', range(count))
Running without the repeat flag:
(env) $ py.test test.py -vv
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items
test.py:4: test_1 PASSED
test.py:8: test_2 PASSED
=========================== 2 passed in 0.01 seconds ===========================
Running with the repeat flag:
(env) $ py.test test.py -vv --repeat 3
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 6 items
test.py:4: test_1[0] PASSED
test.py:4: test_1[1] PASSED
test.py:4: test_1[2] PASSED
test.py:8: test_2[0] PASSED
test.py:8: test_2[1] PASSED
test.py:8: test_2[2] PASSED
=========================== 6 passed in 0.01 seconds ===========================
Further reading:
Based on Frank T’s suggestion, I found a very simple solution in the pytest_generate_tests() callout:
parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')
def pytest_generate_tests (metafunc):
for i in range (metafunc.config.option.count):
metafunc.addcall()
Now executing py.test --count 5 causes each test to be executed five times in the test session.
And it requires no changes to any of our existing tests.
The pytest module pytest-repeat exists for this purpose, and I recommend using modules where possible, rather than re-implementing their functionality yourself.
To use it simply add pytest-repeat to your requirements.txt or pip install pytest-repeat, then execute your tests with --count n.
Based on what I’ve seen here, and given that I already do some filtering of tests in pytest_collection_modifyitems, my method of choice is the following. In conftest.py
def pytest_addoption(parser):
parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')
def pytest_collection_modifyitems(session, config, items):
count = config.option.count
items[:] = items * count # add each test multiple times
While pytest-repeat (the most popular answer) doesn’t work for unittest class tests, pytest-flakefinder does:
pip install pytest-flakefinder
pytest --flake-finder --flake-runs=5 tests...
Before finding test-flakefinder, I wrote a little hack of a script that does a similar thing. You can find it here. The top of the script includes instructions to how it can be run.
I’ve been looking for a simple solution for a long time. I just had to run one test n-number of times. He shouldn’t have fallen once. I solved this problem as simply and stupidly as possible, but it worked for me.
def run_multiple_times(number):
i = 0
while i < number:
#test
i += 1
run_multiple_times(5)
I want to run each selected py.test item an arbitrary number of times, sequentially.
I don’t see any standard py.test mechanism for doing this.
I attempted to do this in the pytest_collection_modifyitems() hook. I modified the list of items passed in, to specify each item more than once. The first execution of a test item works as expected, but that seems to cause some problems for my code.
Further, I would prefer to have a unique test item object for each run, as I use id (item) as a key in various reporting code. Unfortunately, I can’t find any py.test code to duplicate a test item, copy.copy() doesn’t work, and copy.deepcopy() gets an exception.
Can anybody suggest a strategy for executing a test multiple times?
One possible strategy is parameterizing the test in question, but not explicitly using the parameter.
For example:
@pytest.mark.parametrize('execution_number', range(5))
def run_multiple_times(execution_number):
assert True
The above test should run five times.
Check out the parametrization documentation: https://pytest.org/latest/parametrize.html
In order to run each test a number of times, we will programmatically parameterize each test as the tests are being generated.
First, let’s add the parser option (include the following in one of your conftest.py’s):
def pytest_addoption(parser):
parser.addoption('--repeat', action='store',
help='Number of times to repeat each test')
Now we add a “pytest_generate_tests” hook. Here is where the magic happens.
def pytest_generate_tests(metafunc):
if metafunc.config.option.repeat is not None:
count = int(metafunc.config.option.repeat)
# We're going to duplicate these tests by parametrizing them,
# which requires that each test has a fixture to accept the parameter.
# We can add a new fixture like so:
metafunc.fixturenames.append('tmp_ct')
# Now we parametrize. This is what happens when we do e.g.,
# @pytest.mark.parametrize('tmp_ct', range(count))
# def test_foo(): pass
metafunc.parametrize('tmp_ct', range(count))
Running without the repeat flag:
(env) $ py.test test.py -vv
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items
test.py:4: test_1 PASSED
test.py:8: test_2 PASSED
=========================== 2 passed in 0.01 seconds ===========================
Running with the repeat flag:
(env) $ py.test test.py -vv --repeat 3
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 6 items
test.py:4: test_1[0] PASSED
test.py:4: test_1[1] PASSED
test.py:4: test_1[2] PASSED
test.py:8: test_2[0] PASSED
test.py:8: test_2[1] PASSED
test.py:8: test_2[2] PASSED
=========================== 6 passed in 0.01 seconds ===========================
Further reading:
Based on Frank T’s suggestion, I found a very simple solution in the pytest_generate_tests() callout:
parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')
def pytest_generate_tests (metafunc):
for i in range (metafunc.config.option.count):
metafunc.addcall()
Now executing py.test --count 5 causes each test to be executed five times in the test session.
And it requires no changes to any of our existing tests.
The pytest module pytest-repeat exists for this purpose, and I recommend using modules where possible, rather than re-implementing their functionality yourself.
To use it simply add pytest-repeat to your requirements.txt or pip install pytest-repeat, then execute your tests with --count n.
Based on what I’ve seen here, and given that I already do some filtering of tests in pytest_collection_modifyitems, my method of choice is the following. In conftest.py
def pytest_addoption(parser):
parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')
def pytest_collection_modifyitems(session, config, items):
count = config.option.count
items[:] = items * count # add each test multiple times
While pytest-repeat (the most popular answer) doesn’t work for unittest class tests, pytest-flakefinder does:
pip install pytest-flakefinder
pytest --flake-finder --flake-runs=5 tests...
Before finding test-flakefinder, I wrote a little hack of a script that does a similar thing. You can find it here. The top of the script includes instructions to how it can be run.
I’ve been looking for a simple solution for a long time. I just had to run one test n-number of times. He shouldn’t have fallen once. I solved this problem as simply and stupidly as possible, but it worked for me.
def run_multiple_times(number):
i = 0
while i < number:
#test
i += 1
run_multiple_times(5)