Import paths – the right way?
Question:
I know there are A LOT of similar or the same questions, but i still cannot understand / find the right way for me to work with modules. Python is my favorite language, and i like everything in it except working with imports: recursive imports (when you try to reference a name that is not yet there), import paths, etc.
So, I have this kind of a project structure:
my_project/
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
Package1 may be used a standalone unit, but is also expected to be imported by package2.
What am i doing now, is that, for example, in package1.module1 i write from package1 import module2, i.e. using full path to imported module. I do this because if i use import module2 — this will not work when the module will be imported from another package (package2). I also cannot use from . import module2 — this will not work when running module1 directly.
OK, so for from package1 import module2 in package1.module1 to work in both cases (when running directly package1.module1 and when importing it from package2) i add these lines at the beginning of package1.module1:
import os, sys
currDir = os.path.dirname(os.path.realpath(__file__))
rootDir = os.path.abspath(os.path.join(currDir, '..'))
if rootDir not in sys.path: # add parent dir to paths
sys.path.append(rootDir)
For me this works, but i feel this is not pythonic. Am i doing something wrong?
Should i, instead, always run package1.module1 from project root? If so, this makes inconvenient to run it from an IDE — i need somehow to set paths in it.
UPDATE: I tried to add a file root.pth to package1 dir with contents of ... But it didn’t work — i guess it’s intended for something else.
CONCLUSIONS:
-
Always use absolute imports: import package1.module1
-
Add a bootstrapper to the root folder to start some of the modules as a standalone script. This solves running the script form an IDE and is a pythonic approach.
On 4/22/07, Brett Cannon wrote:
This PEP is to change the if __name__ == "__main__": ... idiom to
if __name__ == sys.main: ... so that you at least have a chance
to execute module in a package that use relative imports.
Ran this PEP past python-ideas. Stopped the discussion there when too
many new ideas were being proposed. =) I have listed all of them in
the Rejected Ideas section, although if overwhelming support for one
comes forward the PEP can shift to one of them.
I’m -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a module’s directory, which I’ve always seen as an
antipattern. To make me change my mind you’d have to convince me that
it isn’t.
Answers:
What is the entry point for your program? Usually the entry point for a program will be at the root of the project. Since it is at the root, all the modules within the root will be importable, provided there is an __init__.py file in them.
So, using your example:
my_project/
main.py
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
main.py would be the entry point for your program. Because the file that is executed as main is automatically put on the PYTHONPATH, both package1 and package2 are available from the top level import.
# in main.py
from package1.module1 import *
from package1.module2 import *
# in package1.module1
import module2
from package2.module1 import *
# in package2.module1 import *
import module2
from package1.module1 import *
Note that in the above, package1 and package2 depend on each other. That should never be the case. But this is just an example of being able to import from anywhere.
main.py doesn’t have to be anything fancy either. It can be very simple:
# main.py
if __name__ == '__main__':
from package1.module1 import SomeClass
SomeClass().start()
The point I’m trying to make, is that if a module needs to be accessible by other modules, that module should be available as a top level import. A module should not attempt to put itself as a top level import (directly on the PYTHONPATH).
It should be the responsibility of the project for ensuring that all imports can be satisfied if the module is included directly in the project. There are two ways to do this. The first is by creating a bootstrapper file such as main.py in the project folder. The other, is by creating a file that adds all relevant paths to PYTHONPATH, that is loaded by any entry points that may exist.
For example:
# setup.py
import sys
def load():
paths = ['/path1/','/path2/','/path3/']
for p in path:
sys.path.insert(0, p)
# entrypoint.py
from setup import load
load()
# continue with program
The main thing to take away, is that a module is not supposed to put itself on the path. The path should be determined automatically by the entry point into the program, or defined explicitly by a setup script that knows where all the relevant modules are.
I generally create each package as an installable package (ie, create a setup.py file), and then install them into a virtualenv just for this project, using pip.
You can even install the using pip -e if they are still under development.
I’m 5 years late here.. but just do export PYTHONPATH=/path1:/path2: (notice trailing “:”) – this way your working directory (from which you run python) will be in the path.
I know there are A LOT of similar or the same questions, but i still cannot understand / find the right way for me to work with modules. Python is my favorite language, and i like everything in it except working with imports: recursive imports (when you try to reference a name that is not yet there), import paths, etc.
So, I have this kind of a project structure:
my_project/
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
Package1 may be used a standalone unit, but is also expected to be imported by package2.
What am i doing now, is that, for example, in package1.module1 i write from package1 import module2, i.e. using full path to imported module. I do this because if i use import module2 — this will not work when the module will be imported from another package (package2). I also cannot use from . import module2 — this will not work when running module1 directly.
OK, so for from package1 import module2 in package1.module1 to work in both cases (when running directly package1.module1 and when importing it from package2) i add these lines at the beginning of package1.module1:
import os, sys
currDir = os.path.dirname(os.path.realpath(__file__))
rootDir = os.path.abspath(os.path.join(currDir, '..'))
if rootDir not in sys.path: # add parent dir to paths
sys.path.append(rootDir)
For me this works, but i feel this is not pythonic. Am i doing something wrong?
Should i, instead, always run package1.module1 from project root? If so, this makes inconvenient to run it from an IDE — i need somehow to set paths in it.
UPDATE: I tried to add a file root.pth to package1 dir with contents of ... But it didn’t work — i guess it’s intended for something else.
CONCLUSIONS:
-
Always use absolute imports:
import package1.module1 -
Add a bootstrapper to the root folder to start some of the modules as a standalone script. This solves running the script form an IDE and is a pythonic approach.
On 4/22/07, Brett Cannon wrote:
This PEP is to change the
if __name__ == "__main__": ...idiom to
if __name__ == sys.main: ...so that you at least have a chance
to execute module in a package that use relative imports.Ran this PEP past python-ideas. Stopped the discussion there when too
many new ideas were being proposed. =) I have listed all of them in
the Rejected Ideas section, although if overwhelming support for one
comes forward the PEP can shift to one of them.
I’m -1 on this and on any other proposed twiddlings of the __main__
machinery. The only use case seems to be running scripts that happen
to be living inside a module’s directory, which I’ve always seen as an
antipattern. To make me change my mind you’d have to convince me that
it isn’t.
What is the entry point for your program? Usually the entry point for a program will be at the root of the project. Since it is at the root, all the modules within the root will be importable, provided there is an __init__.py file in them.
So, using your example:
my_project/
main.py
package1/
__init__.py
module1
module2
package2/
__init__.py
module1
module2
main.py would be the entry point for your program. Because the file that is executed as main is automatically put on the PYTHONPATH, both package1 and package2 are available from the top level import.
# in main.py
from package1.module1 import *
from package1.module2 import *
# in package1.module1
import module2
from package2.module1 import *
# in package2.module1 import *
import module2
from package1.module1 import *
Note that in the above, package1 and package2 depend on each other. That should never be the case. But this is just an example of being able to import from anywhere.
main.py doesn’t have to be anything fancy either. It can be very simple:
# main.py
if __name__ == '__main__':
from package1.module1 import SomeClass
SomeClass().start()
The point I’m trying to make, is that if a module needs to be accessible by other modules, that module should be available as a top level import. A module should not attempt to put itself as a top level import (directly on the PYTHONPATH).
It should be the responsibility of the project for ensuring that all imports can be satisfied if the module is included directly in the project. There are two ways to do this. The first is by creating a bootstrapper file such as main.py in the project folder. The other, is by creating a file that adds all relevant paths to PYTHONPATH, that is loaded by any entry points that may exist.
For example:
# setup.py
import sys
def load():
paths = ['/path1/','/path2/','/path3/']
for p in path:
sys.path.insert(0, p)
# entrypoint.py
from setup import load
load()
# continue with program
The main thing to take away, is that a module is not supposed to put itself on the path. The path should be determined automatically by the entry point into the program, or defined explicitly by a setup script that knows where all the relevant modules are.
I generally create each package as an installable package (ie, create a setup.py file), and then install them into a virtualenv just for this project, using pip.
You can even install the using pip -e if they are still under development.
I’m 5 years late here.. but just do export PYTHONPATH=/path1:/path2: (notice trailing “:”) – this way your working directory (from which you run python) will be in the path.