pytest finding and not finding modules to test
Question:
I was going to post this question in code review because rather than a solution I wanted for python experts to check my code.
But while preparing the code I found some related issue so I finally decided to ask it here since it is not more just a code check
My question is how does pytest finds modules to test?
Let me explain
Situation 1
First I have this code structure
|
|-tests
| |----test_sum.py
|
|--script_to_test.py
test_sum.py
import script_to_test
def test_sum():
d=script_to_test.sum(6,5)
assert d==11
script_to_test.py
def sum(a,b):
return a+b
If I create a virtual environment and install pytest there, or if I use a conda environment with pytest installed I can do
pytest
and I will get
pytest
====================================================================== test session starts ======================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/me/pytestExample
collected 1 item
tests/test_sum.py . [100%]
======================================================================= 1 passed in 0.01s =======================================================================
Situation 2
If I add a file pytest.ini in the root
[pytest]
python_files = test_*
python_classes = *Tests
python_functions = test_*
pytest will not work anymore and I will get
pytest
========================================================= test session starts ==========================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/me/pytestExample, configfile: pytest.ini
collected 0 items / 1 error
================================================================ ERRORS ================================================================
__________________________________________________ ERROR collecting tests/test_sum.py __________________________________________________
ImportError while importing test module '/home/me/pytestExample/tests/test_sum.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.9.16/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_sum.py:1: in <module>
import script_to_test
E ModuleNotFoundError: No module named 'script_to_test'
======================================================= short test summary info ========================================================
ERROR tests/test_sum.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================== 1 error in 0.07s ===========================================================
Now, I can solve this already, but the purpose of this question is to know why it fails to work when a pytest.ini is added? and why does it work in the first place when that file is not present?
Answers:
following your structure:
from .. import script_to_test
def test_sum():
d=script_to_test.sum(6,5)
assert d==11
As you are importing from a folder, when you put:
import script_to_test
this will work for every file you have inside test folder…but your script_to_test is not in the same folder…is one level before…so two dots (..) will bring you to the correct level related to this file.
When you run pytest without a pytest.ini file, it searches for all the files in the current directory that start with the word "test" and end with the file extension .py, which is the default behavior for python_files option in pytest. It then collects all the test functions in those files and runs them.
In your first situation, you have a tests directory with a test_sum.py file, which is discovered by pytest because it matches the default pattern. The test_sum.py file imports the script_to_test module and runs a test function.
In your second situation, you have a pytest.ini file in the root directory with some configuration options. The python_files option tells pytest to only collect files that start with "test" and end with "Tests.py" or ".py". Since your script_to_test.py file does not match this pattern, pytest does not collect it as a test file, and when test_sum.py tries to import it, it fails with a ModuleNotFoundError.
To fix this, you can either remove the pytest.ini file or update the python_files option to include script_to_test.py:
[pytest]
python_files = test_*.py script_to_test.py
This tells pytest to collect all files that start with "test" and end with ".py", as well as the script_to_test.py file. With this configuration, pytest should discover and run your tests as expected.
The default glob patterns for test filename discovery are:
["test_*.py", "*_test.py"]
This is documented here and you can see it in the pytest source here.
That means that pytest considers your file script_to_test.py as a test file itself, since it’s matching the second pattern *_test.py. If you add a function named test_something() inside that script_to_test.py file, it will also be called during the test run.
During the test discovery process, pytest will import all the test modules, which makes them available in sys.modules. When your test actually runs, the import statement in test_sum.py is able to resolve the import because script_to_test has already been imported (it’s cached in sys.modules). You could add this to the top of test_sum.py to verify:
import sys
if "script_to_test" in sys.modules:
print("It's already imported!n")
...
Run with pytest -s to show the print output.
collecting ...
It's already imported!
collected 1 item
tests/test_sum.py .
So, the import resolves because of an accident – the code under test is accidentally matching the test filename pattern. If you rename the module to something that’s not matching the default discovery patterns, such as script_to_taste.py, then the test run will fail. Similarly, when you override the default patterns by adding python_files = test_* in pytest.ini you’re disabling the *_test.py pattern and preventing script_to_test.py from being imported during the test discovery process.
I was going to post this question in code review because rather than a solution I wanted for python experts to check my code.
But while preparing the code I found some related issue so I finally decided to ask it here since it is not more just a code check
My question is how does pytest finds modules to test?
Let me explain
Situation 1
First I have this code structure
|
|-tests
| |----test_sum.py
|
|--script_to_test.py
test_sum.py
import script_to_test
def test_sum():
d=script_to_test.sum(6,5)
assert d==11
script_to_test.py
def sum(a,b):
return a+b
If I create a virtual environment and install pytest there, or if I use a conda environment with pytest installed I can do
pytest
and I will get
pytest
====================================================================== test session starts ======================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/me/pytestExample
collected 1 item
tests/test_sum.py . [100%]
======================================================================= 1 passed in 0.01s =======================================================================
Situation 2
If I add a file pytest.ini in the root
[pytest]
python_files = test_*
python_classes = *Tests
python_functions = test_*
pytest will not work anymore and I will get
pytest
========================================================= test session starts ==========================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/me/pytestExample, configfile: pytest.ini
collected 0 items / 1 error
================================================================ ERRORS ================================================================
__________________________________________________ ERROR collecting tests/test_sum.py __________________________________________________
ImportError while importing test module '/home/me/pytestExample/tests/test_sum.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.9.16/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_sum.py:1: in <module>
import script_to_test
E ModuleNotFoundError: No module named 'script_to_test'
======================================================= short test summary info ========================================================
ERROR tests/test_sum.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================== 1 error in 0.07s ===========================================================
Now, I can solve this already, but the purpose of this question is to know why it fails to work when a pytest.ini is added? and why does it work in the first place when that file is not present?
following your structure:
from .. import script_to_test
def test_sum():
d=script_to_test.sum(6,5)
assert d==11
As you are importing from a folder, when you put:
import script_to_test
this will work for every file you have inside test folder…but your script_to_test is not in the same folder…is one level before…so two dots (..) will bring you to the correct level related to this file.
When you run pytest without a pytest.ini file, it searches for all the files in the current directory that start with the word "test" and end with the file extension .py, which is the default behavior for python_files option in pytest. It then collects all the test functions in those files and runs them.
In your first situation, you have a tests directory with a test_sum.py file, which is discovered by pytest because it matches the default pattern. The test_sum.py file imports the script_to_test module and runs a test function.
In your second situation, you have a pytest.ini file in the root directory with some configuration options. The python_files option tells pytest to only collect files that start with "test" and end with "Tests.py" or ".py". Since your script_to_test.py file does not match this pattern, pytest does not collect it as a test file, and when test_sum.py tries to import it, it fails with a ModuleNotFoundError.
To fix this, you can either remove the pytest.ini file or update the python_files option to include script_to_test.py:
[pytest]
python_files = test_*.py script_to_test.py
This tells pytest to collect all files that start with "test" and end with ".py", as well as the script_to_test.py file. With this configuration, pytest should discover and run your tests as expected.
The default glob patterns for test filename discovery are:
["test_*.py", "*_test.py"]
This is documented here and you can see it in the pytest source here.
That means that pytest considers your file script_to_test.py as a test file itself, since it’s matching the second pattern *_test.py. If you add a function named test_something() inside that script_to_test.py file, it will also be called during the test run.
During the test discovery process, pytest will import all the test modules, which makes them available in sys.modules. When your test actually runs, the import statement in test_sum.py is able to resolve the import because script_to_test has already been imported (it’s cached in sys.modules). You could add this to the top of test_sum.py to verify:
import sys
if "script_to_test" in sys.modules:
print("It's already imported!n")
...
Run with pytest -s to show the print output.
collecting ...
It's already imported!
collected 1 item
tests/test_sum.py .
So, the import resolves because of an accident – the code under test is accidentally matching the test filename pattern. If you rename the module to something that’s not matching the default discovery patterns, such as script_to_taste.py, then the test run will fail. Similarly, when you override the default patterns by adding python_files = test_* in pytest.ini you’re disabling the *_test.py pattern and preventing script_to_test.py from being imported during the test discovery process.