Why doesn't my current directory show up in the path using pytest on Windows?
Question:
I have the following folder structure;
myapp
myapp
__init__.py
tests
test_myapp.py
and my pwd is
C:Userswwernerprogrammingmyapp
I have the following test setup:
import sys
import pprint
def test_cool():
pprint.pprint(sys.path)
assert False
That produces the following paths:
['C:\Users\wwerner\programming\myapp\tests',
'C:\Users\wwerner\programming\envs\myapp\Scripts',
'C:\Windows\system32\python34.zip',
'C:\Python34\DLLs',
'C:\Python34\lib',
'C:\Python34',
'C:\Users\wwerner\programming\envs\myapp',
'C:\Users\wwerner\programming\envs\myapp\lib\site-packages']
And when I try to import myapp I get the following error:
ImportError: No module named 'myapp'
So it looks like it’s not adding the current directory to my path.
By changing my import line to look like this:
import sys
sys.path.insert(0, '.')
import myapp
I am then able to import myapp with no problems.
Why does my current directory not show up in the path when running pytest? Is my only workaround to insert . into the sys.path? (I’m using Python 3.4 if it matters)
Answers:
Use the environment variable PYTHONPATH.
In Windows:
set PYTHONPATH=.
py.test
In Unix:
PYTHONPATH=. py.test
sys.path automatically has the script’s directory in it, and not the current working directory.
I am guessing that your script in placed in tests directory. Based on this assumption, your code should look like this:
import sys
import os
ROOT_DIR = os.path.dirname(os.path.dirname(__file__))
sys.path.append(ROOT_DIR)
import myapp # Should work now
Ahah!
After comparing the layout of my cookiecutter repo, it turns out to be way more simple (and better) than that.
tests/
__init__.py
test_myapp.py
A simple addition of the __init__.py file to my test dir allows me to run py.test from my main directory.
Using an installable package
If you have an installable package (setup.py or pyproject.toml file with a build-system defined) then it’s best to test against the installed code. Installing the code in a venv will make import statements resolve correctly within that venv.
pip install --editable .
pytest
The simplest possible way to make the project shown in the question into an installable package would be by adding this setup.py:
from setuptools import setup
setup(
name="myapp",
version="0.1",
packages=["myapp"],
)
This will put the myapp code at /path/to/myapp/.venv/lib/python3.XY/site-packages, which is in the sys.path of the virtual environment. Now myapp can be imported from the site-packages dir, just as it would be for a user installation. It is neither necessary nor desirable for the current working directory to be present on sys.path during test execution.
Not using an installable package
The project shown in the question does not have any installer, so it can’t be installed. It can still be tested by making sure the project root (i.e. the directory which contains both myapp and tests as subdirectories) is present on sys.path.
The best way to do this is to use python -m pytest, rather than invoking the bare pytest command. When you use python -m pytest it adds the current working directory to the start of sys.path. That’s the normal Python behavior when executing a package as __main__ (documented here) and it’s also a documented usage for pytest – see Invoking pytest versus python -m pytest.
Why does adding an __init__.py to the tests subdirectory (not) work?
The directory structure shown in the question is the "Tests outside application code" pattern, documented here. This is also the directory structure I recommend, since it creates a clear distinction between library/application code and test code.
It’s not recommended to add __init__.py files inside the test directories when using a "Tests outside application code" structure, since the test files aren’t intended to be "packaged" (e.g. test files do not really need to import from other test files, and they do not need to be installed at all for end users of your package).
The reason adding a myapp/__init__.py actually allows myapp to be imported by pytest (as shown in Wayne’s answer) is actually an accident due to the way test discovery appends sys.path during the test collection phase. This is described as "problematic" in the docs
… this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable
They go on to strongly recommend using the src-layout if you intend to have __init__.py files inside test directories, to avoid this confusion of the import system.
But perhaps the best reason not to rely on this side-effect is that pytest collection actually can work in multiple modes (see import modes), and Wayne’s answer relies upon pytest using the "prepend" import mode. Prepend mode is currently the default, but the docs mention that a future version will switch to using "importlib" mode by default:
We intend to make importlib the default in future releases.
The accepted answer does not work with pytest --import-mode=importlib and so will stop working altogether at some stage.
I have the following folder structure;
myapp
myapp
__init__.py
tests
test_myapp.py
and my pwd is
C:Userswwernerprogrammingmyapp
I have the following test setup:
import sys
import pprint
def test_cool():
pprint.pprint(sys.path)
assert False
That produces the following paths:
['C:\Users\wwerner\programming\myapp\tests',
'C:\Users\wwerner\programming\envs\myapp\Scripts',
'C:\Windows\system32\python34.zip',
'C:\Python34\DLLs',
'C:\Python34\lib',
'C:\Python34',
'C:\Users\wwerner\programming\envs\myapp',
'C:\Users\wwerner\programming\envs\myapp\lib\site-packages']
And when I try to import myapp I get the following error:
ImportError: No module named 'myapp'
So it looks like it’s not adding the current directory to my path.
By changing my import line to look like this:
import sys
sys.path.insert(0, '.')
import myapp
I am then able to import myapp with no problems.
Why does my current directory not show up in the path when running pytest? Is my only workaround to insert . into the sys.path? (I’m using Python 3.4 if it matters)
Use the environment variable PYTHONPATH.
In Windows:
set PYTHONPATH=.
py.test
In Unix:
PYTHONPATH=. py.test
sys.path automatically has the script’s directory in it, and not the current working directory.
I am guessing that your script in placed in tests directory. Based on this assumption, your code should look like this:
import sys
import os
ROOT_DIR = os.path.dirname(os.path.dirname(__file__))
sys.path.append(ROOT_DIR)
import myapp # Should work now
Ahah!
After comparing the layout of my cookiecutter repo, it turns out to be way more simple (and better) than that.
tests/
__init__.py
test_myapp.py
A simple addition of the __init__.py file to my test dir allows me to run py.test from my main directory.
Using an installable package
If you have an installable package (setup.py or pyproject.toml file with a build-system defined) then it’s best to test against the installed code. Installing the code in a venv will make import statements resolve correctly within that venv.
pip install --editable .
pytest
The simplest possible way to make the project shown in the question into an installable package would be by adding this setup.py:
from setuptools import setup
setup(
name="myapp",
version="0.1",
packages=["myapp"],
)
This will put the myapp code at /path/to/myapp/.venv/lib/python3.XY/site-packages, which is in the sys.path of the virtual environment. Now myapp can be imported from the site-packages dir, just as it would be for a user installation. It is neither necessary nor desirable for the current working directory to be present on sys.path during test execution.
Not using an installable package
The project shown in the question does not have any installer, so it can’t be installed. It can still be tested by making sure the project root (i.e. the directory which contains both myapp and tests as subdirectories) is present on sys.path.
The best way to do this is to use python -m pytest, rather than invoking the bare pytest command. When you use python -m pytest it adds the current working directory to the start of sys.path. That’s the normal Python behavior when executing a package as __main__ (documented here) and it’s also a documented usage for pytest – see Invoking pytest versus python -m pytest.
Why does adding an __init__.py to the tests subdirectory (not) work?
The directory structure shown in the question is the "Tests outside application code" pattern, documented here. This is also the directory structure I recommend, since it creates a clear distinction between library/application code and test code.
It’s not recommended to add __init__.py files inside the test directories when using a "Tests outside application code" structure, since the test files aren’t intended to be "packaged" (e.g. test files do not really need to import from other test files, and they do not need to be installed at all for end users of your package).
The reason adding a myapp/__init__.py actually allows myapp to be imported by pytest (as shown in Wayne’s answer) is actually an accident due to the way test discovery appends sys.path during the test collection phase. This is described as "problematic" in the docs
… this introduces a subtle problem: in order to load the test modules from the
testsdirectory, pytest prepends the root of the repository tosys.path, which adds the side-effect that nowmypkgis also importable
They go on to strongly recommend using the src-layout if you intend to have __init__.py files inside test directories, to avoid this confusion of the import system.
But perhaps the best reason not to rely on this side-effect is that pytest collection actually can work in multiple modes (see import modes), and Wayne’s answer relies upon pytest using the "prepend" import mode. Prepend mode is currently the default, but the docs mention that a future version will switch to using "importlib" mode by default:
We intend to make
importlibthe default in future releases.
The accepted answer does not work with pytest --import-mode=importlib and so will stop working altogether at some stage.