Using pytest with a src layer
Question:
pytest recommends including an additional directory to separate the source code within a project:
my_package
├── src # <-- no __init__.py on this layer
│ └── my_package
│ ├── __init__.py
│ └── util_module
│ ├── __init__.py
│ └── utils.py
└── tests
├── __init__.py
└── test_util_module
├── __init__.py
└── test_utils.py
Sadly, they say nothing[1] about how imports in the test code should work in such a case, which work for my IDE just fine in this naive example[2], but causes the following error with pytest:
~/my_package$ pytest
====================== test session starts ======================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/user/workspace/my_package, inifile:
collected 0 items / 1 errors
============================ ERRORS =============================
___ ERROR collecting tests/test_util_module/test_utils.py ___
ImportError while importing test module '/home/user/workspace/my_package/tests/test_util_module/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_util_module/test_utils.py:1: in <module>
from test.test_module.some_file import starify
E ModuleNotFoundError: No module named 'my_package.util_module'
!!!! Interrupted: 1 errors during collection !!!!!
I can fix the issue by changing the import of the test to
from src.my_package.util_module.utils import starify
but then my IDE complaints about the src
part being redundant, so I’d like to keep it out.
[1]: Not the case any more. As of version 3.7.3, pytest recommends the editable install also featured in @hoefling’s answer at the top of its good practices.
[2]: Setup is virtualenv env -p python3.6; source env/bin/activate; pip install pytest
Answers:
Recommended approach for pytest>=7
: use the pythonpath
setting
Recently, pytest
has added a new core plugin that supports sys.path
modifications via the pythonpath
configuration value. The solution is thus much simpler now and doesn’t require any workarounds anymore:
pyproject.toml
example:
[tool.pytest.ini_options]
pythonpath = [
"src"
]
pytest.ini
example:
[pytest]
pythonpath = src
The path entries are calculated relative to the rootdir, thus the src
entry adds path/to/project/src
directory to sys.path
in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── src2/
| └── lib2.py
└── tests
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
"src", "src2",
]
or
[pytest]
pythonpath = src src2
will add both lib
and lib2
modules to sys.path
, so
import lib
import lib2
will both work.
Original answer
Adjusting the PYTHONPATH
(as suggested in the comments) is one possibility to solve the import issue. Another is adding an empty conftest.py
file in the src
directory:
$ touch src/conftest.py
and pytest
will add src
to sys.path
. This is a simple way to trick pytest
into adding codebase to sys.path
.
However, the src
layout is usually selected when you intend to build a distribution, e.g. providing a setup.py
with (in this case) explicitly specifying the root package dir:
from setuptools import find_packages, setup
setup(
...
package_dir={'': 'src'},
packages=find_packages(where='src'),
...
)
and installing the package in the development mode (via python setup.py develop
or pip install --editable .
) while you’re still developing it. This way, your package my_package
is correctly integrated in the Python’s site packages structure and there’s no need to fiddle with PYTHONPATH
.
PYTHONPATH updates weren’t working for me when using github actions (known prob). Using this pytest-pythonpath install with pytest.ini file worked for me instead:
pip install pytest-pythonpath # accompany with python_path in pytest.ini, so PYTHONPATH is updated with location for modules under test
With this, basic ‘pytest’ command happily found all tests in subdirs, and found modules under test based on my pytest.ini (set to match source folders in pycharm)
As of PyTest 7.0.0, you can now use the pythonpath
option to set some default entries in sys.path
. This is the most convenient way of going about this.
In pytest.ini
:
[pytest]
pythonpath = src/
In pyproject.toml
:
[tool.pytest.ini_options]
pythonpath = "src/"
If you use tox
, then you should unset this in its settings, so you don’t accidentally test the local version, rather than the installed version.
In tox.ini
:
[testenv]
commands = pytest ... -o pythonpath=
See the documentation: https://docs.pytest.org/en/7.1.x/reference/reference.html#confval-pythonpath
pytest recommends including an additional directory to separate the source code within a project:
my_package
├── src # <-- no __init__.py on this layer
│ └── my_package
│ ├── __init__.py
│ └── util_module
│ ├── __init__.py
│ └── utils.py
└── tests
├── __init__.py
└── test_util_module
├── __init__.py
└── test_utils.py
Sadly, they say nothing[1] about how imports in the test code should work in such a case, which work for my IDE just fine in this naive example[2], but causes the following error with pytest:
~/my_package$ pytest
====================== test session starts ======================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/user/workspace/my_package, inifile:
collected 0 items / 1 errors
============================ ERRORS =============================
___ ERROR collecting tests/test_util_module/test_utils.py ___
ImportError while importing test module '/home/user/workspace/my_package/tests/test_util_module/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_util_module/test_utils.py:1: in <module>
from test.test_module.some_file import starify
E ModuleNotFoundError: No module named 'my_package.util_module'
!!!! Interrupted: 1 errors during collection !!!!!
I can fix the issue by changing the import of the test to
from src.my_package.util_module.utils import starify
but then my IDE complaints about the src
part being redundant, so I’d like to keep it out.
[1]: Not the case any more. As of version 3.7.3, pytest recommends the editable install also featured in @hoefling’s answer at the top of its good practices.
[2]: Setup is virtualenv env -p python3.6; source env/bin/activate; pip install pytest
Recommended approach for pytest>=7
: use the pythonpath
setting
Recently, pytest
has added a new core plugin that supports sys.path
modifications via the pythonpath
configuration value. The solution is thus much simpler now and doesn’t require any workarounds anymore:
pyproject.toml
example:
[tool.pytest.ini_options]
pythonpath = [
"src"
]
pytest.ini
example:
[pytest]
pythonpath = src
The path entries are calculated relative to the rootdir, thus the src
entry adds path/to/project/src
directory to sys.path
in this case.
Multiple path entries are also allowed: for a layout
repo/
├── src/
| └── lib.py
├── src2/
| └── lib2.py
└── tests
└── test_lib.py
the configuration
[tool.pytest.ini_options]
pythonpath = [
"src", "src2",
]
or
[pytest]
pythonpath = src src2
will add both lib
and lib2
modules to sys.path
, so
import lib
import lib2
will both work.
Original answer
Adjusting the PYTHONPATH
(as suggested in the comments) is one possibility to solve the import issue. Another is adding an empty conftest.py
file in the src
directory:
$ touch src/conftest.py
and pytest
will add src
to sys.path
. This is a simple way to trick pytest
into adding codebase to sys.path
.
However, the src
layout is usually selected when you intend to build a distribution, e.g. providing a setup.py
with (in this case) explicitly specifying the root package dir:
from setuptools import find_packages, setup
setup(
...
package_dir={'': 'src'},
packages=find_packages(where='src'),
...
)
and installing the package in the development mode (via python setup.py develop
or pip install --editable .
) while you’re still developing it. This way, your package my_package
is correctly integrated in the Python’s site packages structure and there’s no need to fiddle with PYTHONPATH
.
PYTHONPATH updates weren’t working for me when using github actions (known prob). Using this pytest-pythonpath install with pytest.ini file worked for me instead:
pip install pytest-pythonpath # accompany with python_path in pytest.ini, so PYTHONPATH is updated with location for modules under test
With this, basic ‘pytest’ command happily found all tests in subdirs, and found modules under test based on my pytest.ini (set to match source folders in pycharm)
As of PyTest 7.0.0, you can now use the pythonpath
option to set some default entries in sys.path
. This is the most convenient way of going about this.
In pytest.ini
:
[pytest]
pythonpath = src/
In pyproject.toml
:
[tool.pytest.ini_options]
pythonpath = "src/"
If you use tox
, then you should unset this in its settings, so you don’t accidentally test the local version, rather than the installed version.
In tox.ini
:
[testenv]
commands = pytest ... -o pythonpath=
See the documentation: https://docs.pytest.org/en/7.1.x/reference/reference.html#confval-pythonpath