Coverage report for 'pytest' different when run in code vs CLI
Question:
When I use the pytest CLI to run some tests against my small module, the coverage report that is output is complete and shows 100% coverage.
However, when I run the tests in code using the pytest module, the coverage report is incomplete (missing the init.py file) and shows some lines as missing. But this cannot be the case, as all of the tests pass.
Looking at the file colours.py, the lines listed as missing are all of the import statements, class definitions, class parameters, method definitions and decorators.
Note that I have compared the verbose output for both runs, and other than the difference in coverage and the odd millisecond of execution time, they are identical.
What could be causing this inconsistent behaviour?
Tests run via the CLI
pytest ./az/tests/unit --cov src --cov-report term-missing
================================ test session starts ================================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/dgard/repos/management-groups/az/tests
configfile: pytest.ini
plugins: cov-4.1.0, dependency-0.5.1
collected 31 items
az/tests/unit/colours_test.py ............................... [100%]
31 assertions pased.
---------- coverage: platform linux, python 3.10.6-final-0 -----------
Name Stmts Miss Cover Missing
--------------------------------------------------
az/src/__init__.py 1 0 100%
az/src/colours.py 35 0 100%
--------------------------------------------------
TOTAL 36 0 100%
================================ 31 passed in 0.10s =================================
Tests run via code
import pytest
pytest_args = ['./az/tests/unit', '--cov', 'src', '--cov-report', 'term-missing']
pytest.main(pytest_args)
================================ test session starts ================================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/dgard/repos/management-groups/az/tests
configfile: pytest.ini
plugins: cov-4.1.0, dependency-0.5.1
collected 31 items
az/tests/unit/colours_test.py ............................... [100%]
31 assertions pased.
---------- coverage: platform linux, python 3.10.6-final-0 -----------
Name Stmts Miss Cover Missing
-------------------------------------------------
az/src/colours.py 35 15 57% 1-18, 42-43
-------------------------------------------------
TOTAL 35 15 57%
================================ 31 passed in 0.09s =================================
Answers:
The issue here seems to be that the script being tested has already been imported by the module that was running pytest.main() in code.
To get around the issue, I am reimporting the script in the tests.
import colours
import importlib
importlib.reload(assets.colours)
When I use the pytest CLI to run some tests against my small module, the coverage report that is output is complete and shows 100% coverage.
However, when I run the tests in code using the pytest module, the coverage report is incomplete (missing the init.py file) and shows some lines as missing. But this cannot be the case, as all of the tests pass.
Looking at the file colours.py, the lines listed as missing are all of the import statements, class definitions, class parameters, method definitions and decorators.
Note that I have compared the verbose output for both runs, and other than the difference in coverage and the odd millisecond of execution time, they are identical.
What could be causing this inconsistent behaviour?
Tests run via the CLI
pytest ./az/tests/unit --cov src --cov-report term-missing
================================ test session starts ================================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/dgard/repos/management-groups/az/tests
configfile: pytest.ini
plugins: cov-4.1.0, dependency-0.5.1
collected 31 items
az/tests/unit/colours_test.py ............................... [100%]
31 assertions pased.
---------- coverage: platform linux, python 3.10.6-final-0 -----------
Name Stmts Miss Cover Missing
--------------------------------------------------
az/src/__init__.py 1 0 100%
az/src/colours.py 35 0 100%
--------------------------------------------------
TOTAL 36 0 100%
================================ 31 passed in 0.10s =================================
Tests run via code
import pytest
pytest_args = ['./az/tests/unit', '--cov', 'src', '--cov-report', 'term-missing']
pytest.main(pytest_args)
================================ test session starts ================================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/dgard/repos/management-groups/az/tests
configfile: pytest.ini
plugins: cov-4.1.0, dependency-0.5.1
collected 31 items
az/tests/unit/colours_test.py ............................... [100%]
31 assertions pased.
---------- coverage: platform linux, python 3.10.6-final-0 -----------
Name Stmts Miss Cover Missing
-------------------------------------------------
az/src/colours.py 35 15 57% 1-18, 42-43
-------------------------------------------------
TOTAL 35 15 57%
================================ 31 passed in 0.09s =================================
The issue here seems to be that the script being tested has already been imported by the module that was running pytest.main() in code.
To get around the issue, I am reimporting the script in the tests.
import colours
import importlib
importlib.reload(assets.colours)