Python project directory structure / pytest trouble
Question:
This should be the easiest problem on earth, but even after extensive searching and tinkering, I’m still in deep trouble with finding a “correct” way to lay a directory structure and manage to run pytest etc correctly.
Let’s say my I have a program called apple.
|- README.md
|- apple
| |-- __init__.py
| |-- apple.py
| - tests
| |-- test_everything.py
The apple.py contains some functions, for examples sake let’s call one eat(). And the test_everything.py file contains some tests like assert eat()=="foobar". So good so easy, but then the fun begins:
- What about the
__init__.py in the apple directory… correct? Empty or what should be inside?
- Is it best practice to call
py.test from the root directory? Or py.test tests?
- So many projects have a
__init__.py in their test directory, but that’s explicitly said to be wrong in the py.test documentation. So why god why
- What comes at the top of the
test_everything.py file: an import apple or from apple import *? or something else entirely
- Do you call the functions then by
eat() or apple.eat()?
- Some even recommend manipulating
os.path.dirname in python
This should be easy, but I’ve seen every combination of the above, not even speaking about tox and the myriad of other tools. Yet with the slightest error, you get thrown some ImportError: No module named 'apple' or some other funky error.
What is the “right” way? The advice and the existing code on github etc follows extremely different conventions. For a medium-experienced coder, this should be much easier.
Answers:
I have arranged an example that works here.
I think naming both package and module apple confusing, perhaps that was one source of confusion. The only non-obvious part IMO is that you must set PYTHONPATH to the current directory if you’re not using a proper setup.py distutils package to install your apple package.
- What about the
__init__.py in the apple directory… correct? Empty or what should be inside?
Yes, correct. Most frequently empty. If you put foo = 42 in it you can later do from apple import foo while you’ll need to do from apple.apple import foo if you put it in apple.py. While it might seem convenient you should use it sparingly.
- Is it best practice to call py.test from the root directory? Or py.test tests?
py.test should be able to find your tests regardless, but see below..
- So many projects have a
__init__.py in their test directory, but that’s explicitly said to be wrong in the py.test documentation. So why god why
So you can import a file in tests that provide common test functionality. In py.test that might be better achieved by creating fixtures in a file called tests/conftest.py.
- What comes at the top of the test_everything.py file: an import apple or from apple import *? or something else entirely
from apple import apple
- Do you call the functions then by eat() or apple.eat()?
apple.eat()
- Some even recommend manipulating os.path.dirname in python
That seems very fragile. I would suggest either
(a) set the environment variable PYTHONPATH to point to the folder where README.md is, or better
(b) create a setup.py file (at the same level as your README.md file), here’s a minimal one:
from setuptools import setup
setup(name='apple', packages=['apple'])
Run the file like so:
python setup.py develop
now apple is globally available and you should never see a no module named apple problem again, i.e. you can run py.test from the root folder or the tests folder.
You can read more about setup.py in the Python Packaging User Guide at https://packaging.python.org/
This should be the easiest problem on earth, but even after extensive searching and tinkering, I’m still in deep trouble with finding a “correct” way to lay a directory structure and manage to run pytest etc correctly.
Let’s say my I have a program called apple.
|- README.md
|- apple
| |-- __init__.py
| |-- apple.py
| - tests
| |-- test_everything.py
The apple.py contains some functions, for examples sake let’s call one eat(). And the test_everything.py file contains some tests like assert eat()=="foobar". So good so easy, but then the fun begins:
- What about the
__init__.pyin the apple directory… correct? Empty or what should be inside? - Is it best practice to call
py.testfrom the root directory? Orpy.test tests? - So many projects have a
__init__.pyin their test directory, but that’s explicitly said to be wrong in the py.test documentation. So why god why - What comes at the top of the
test_everything.pyfile: animport appleorfrom apple import *? or something else entirely - Do you call the functions then by
eat()orapple.eat()? - Some even recommend manipulating
os.path.dirnamein python
This should be easy, but I’ve seen every combination of the above, not even speaking about tox and the myriad of other tools. Yet with the slightest error, you get thrown some ImportError: No module named 'apple' or some other funky error.
What is the “right” way? The advice and the existing code on github etc follows extremely different conventions. For a medium-experienced coder, this should be much easier.
I have arranged an example that works here.
I think naming both package and module apple confusing, perhaps that was one source of confusion. The only non-obvious part IMO is that you must set PYTHONPATH to the current directory if you’re not using a proper setup.py distutils package to install your apple package.
- What about the
__init__.pyin the apple directory… correct? Empty or what should be inside?
Yes, correct. Most frequently empty. If you put foo = 42 in it you can later do from apple import foo while you’ll need to do from apple.apple import foo if you put it in apple.py. While it might seem convenient you should use it sparingly.
- Is it best practice to call py.test from the root directory? Or py.test tests?
py.test should be able to find your tests regardless, but see below..
- So many projects have a
__init__.pyin their test directory, but that’s explicitly said to be wrong in the py.test documentation. So why god why
So you can import a file in tests that provide common test functionality. In py.test that might be better achieved by creating fixtures in a file called tests/conftest.py.
- What comes at the top of the test_everything.py file: an import apple or from apple import *? or something else entirely
from apple import apple
- Do you call the functions then by eat() or apple.eat()?
apple.eat()
- Some even recommend manipulating os.path.dirname in python
That seems very fragile. I would suggest either
(a) set the environment variable PYTHONPATH to point to the folder where README.md is, or better
(b) create a setup.py file (at the same level as your README.md file), here’s a minimal one:
from setuptools import setup
setup(name='apple', packages=['apple'])
Run the file like so:
python setup.py develop
now apple is globally available and you should never see a no module named apple problem again, i.e. you can run py.test from the root folder or the tests folder.
You can read more about setup.py in the Python Packaging User Guide at https://packaging.python.org/