How to organize a Python Project?

From a file system perspective, a module is a file ending with .py and a package is a folder containing modules and (nested) packages again. Python recognizes a folder as a package if it contains a __init__.py file.

A file structure like that

some/
    __init__.py
    foofoo.py
    thing/
        __init__.py
        barbar.py

defines the package some, which has a module foofoo and a nested package thing, which again has a module barbar. However, when using packages and modules, you don’t really distinguish these two types:

import some

some.dothis() # dothis is defined in 'some/__init__.py'

import some.foofoo # <- module
import some.thing # <- package

Please follow PEP8 when selecting naming your packages/modules (i.e. use lower-case names).

A package is a directory with a __init__.py in it. The difference from a directory is that you can import it.

There isn’t a “Python way” per se, but you’ll find that it’s a good idea to put all your modules in one package with a name related to the project.

Also, to follow the Python style guide, PEP8, the package and module names should be all lowercase. So, if we assume the project is called “Botond Statistics” your structure would be something like this:

botondstats/
    indicators/
        moving_averages.py
        stochastics.py
    strategies/
        moving_averages_cross.py
    example.py

You would then find the Stochastics class by doing

from botondstats.indicators.stochastics.Stochastics

(There are various ways to keep the structure but make imports shorter, but that’s another question).

You can put this structure under src/ if you want to, but it’s not necessary. I never do.
Instead I have a main directory:

BotondStatistics/
    docs/
    botonstats/ # the above structure
    setup.py    # Distutils/distribute configuration for packaging.

In this directory I also typically have a virtualenv so I actually also have bin/ lib/ et al. Development is typically done by running

./bin/python setup.py tests

As I use the Distrubute test runner to run the tests.

That’s how I do it. 🙂

You might want to check out the modern-package-template libary. It provides a way to setup a really nice basic layout for a project that walks you through a few questions and tries to help you get something that’s able to be distributed fairly easily.

http://pypi.python.org/pypi/modern-package-template

See python-package-template

Directory structure

    .
    |-- bin
    |   `-- my_program
    |-- docs
    |   `-- doc.txt
    |-- my_program
    |   |-- data
    |   |   `-- some_data.html
    |   |-- __init__.py
    |   |-- submodule
    |   |   `-- __init__.py
    |   |-- helpers.py
    |-- tests
    |   |-- __init__.py
    |   |-- test_helpers.py
    |-- Makefile
    |-- CHANGES.txt
    |-- LICENSE.txt
    |-- README.md
    |-- requirements-dev.txt
    |-- requirements.txt
    `-- setup.py

cat Makefile

    PYTHON=`which python`
    NAME=`python setup.py --name`


    all: check test source deb

    init:
        pip install -r requirements.txt --use-mirrors

    dist: source deb

    source:
        $(PYTHON) setup.py sdist

    deb:
        $(PYTHON) setup.py --command-packages=stdeb.command bdist_deb

    rpm:
        $(PYTHON) setup.py bdist_rpm --post-install=rpm/postinstall --pre-uninstall=rpm/preuninstall

    test:
        unit2 discover -s tests -t .
        python -mpytest weasyprint

    check:
        find . -name *.py | grep -v "^test_" | xargs pylint --errors-only --reports=n
        # pep8
        # pyntch
        # pyflakes
        # pychecker
        # pymetrics

    clean:
        $(PYTHON) setup.py clean
        rm -rf build/ MANIFEST dist build my_program.egg-info deb_dist
        find . -name '*.pyc' -delete

Try python_boilerplate_template:

https://pypi.python.org/pypi/python_boilerplate_template

Before deciding on a project structure, it’s good to ask yourself what the purpose of the project is going to be. Is this going to be one off analysis? A toy concept you want to investigate? A full blown project you intend to distribute? The amount of effort you want to put into structuring your project will be different.

• If it’s a one off analysis, I like to use ipython notebooks. The notebook will capture the flow of your thoughts, and you can add notes in markup to your code for later reference.
• If it’s a toy concept you want to investigate, I find a simple, quick approach to work best. You want to be able to quickly implement your concept to discover if it’s even feasible and thus worth spending more time on it. Part of Python’s philosophy is ‘Don’t try for perfection because “good enough” is often just that.’ You can always come back later and structure your project in a way that follows best software engineering practices.

• If you want to structure your project so you can later distribute it, and so that it scales to many modules I recommend the following structure:

  projectname
   ├── MANIFEST.in
   ├── setup.py
   ├── README
   ├── .gitignore
   ├── .git
   ├── projectname_env
   └── projectname
       ├── __init__.py
       ├── subpackageone
       │   ├── __init__.py
       │   ├── second_module.py
       │   ├── tests
       │   │   └── test_second_module.py
       │   └── models
       │       └── model1
       ├── first_module.py   
       └── tests
           └── test_second_module.py
  

The detailed reasons why I like this structure are in my blog post, but the basic gist is that the hierarchically lower level projectname directory contains your actual project. Alongside it are all the tools that help manage (git) and package (setup.py, MANIFEST.in) it.

The cookiecutter project by audreyr includes several Python project templates:

• https://github.com/audreyr/cookiecutter#python

The package uses a single ~/.cookiecutterrc file to create custom project templates in Python, Java, JS, and other languages.

For example, a Python template compatible with PyPI:

• https://github.com/audreyr/cookiecutter-pypackage