nightmare with relative imports, how does pep 366 work?
Question:
I have a “canonical file structure” like that (I’m giving sensible names to ease the reading):
mainpack/
__main__.py
__init__.py
- helpers/
__init__.py
path.py
- network/
__init__.py
clientlib.py
server.py
- gui/
__init__.py
mainwindow.py
controllers.py
In this structure, for example modules contained in each package may want to access the helpers utilities through relative imports in something like:
# network/clientlib.py
from ..helpers.path import create_dir
The program is runned “as a script” using the __main__.py file in this way:
python mainpack/
Trying to follow the PEP 366 I’ve put in __main__.py these lines:
___package___ = "mainpack"
from .network.clientlib import helloclient
But when running:
$ python mainpack
Traceback (most recent call last):
File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "path/mainpack/__main__.py", line 2, in <module>
from .network.clientlib import helloclient
SystemError: Parent module 'mainpack' not loaded, cannot perform relative import
What’s wrong? What is the correct way to handle and effectively use relative imports?
I’ve tried also to add the current directory to the PYTHONPATH, nothing changes.
Answers:
The loading code seems to be something like this:
try:
return sys.modules[pkgname]
except KeyError:
if level < 1:
warn("Parent module '%s' not found while handling "
"absolute import" % pkgname, RuntimeWarning, 1)
return None
else:
raise SystemError, ("Parent module '%s' not loaded, cannot "
"perform relative import" % pkgname)
which makes me think that maybe your module is not on sys.path. If you start Python (normally) and just type “import mainpack” on the prompt, what does it do? It should be able to find it.
I have tried it myself and got the same error. After reading a bit I found the following solution:
# foo/__main__.py
import sys
mod = __import__('foo')
sys.modules["foo"]=mod
__package__='foo'
from .bar import hello
hello()
It seems a bit hackish to me but it does work. The trick seems to be making sure package foo is loaded so the import can be relative.
The “boilerplate” given in PEP 366 seems incomplete. Although it sets the __package__ variable, it doesn’t actually import the package, which is also needed to allow relative imports to work. extraneon‘s solution is on the right track.
Note that it is not enough to simply have the directory containing the module in sys.path, the corresponding package needs to be explicitly imported. The following seems like a better boilerplate than what was given in PEP 366 for ensuring that a python module can be executed regardless of how it is invoked (through a regular import, or with python -m, or with python, from any location):
# boilerplate to allow running as script directly
if __name__ == "__main__" and __package__ is None:
import sys, os
# The following assumes the script is in the top level of the package
# directory. We use dirname() to help get the parent directory to add to
# sys.path, so that we can import the current package. This is necessary
# since when invoked directly, the 'current' package is not automatically
# imported.
parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(1, parent_dir)
import mypackage
__package__ = str("mypackage")
del sys, os
# now you can use relative imports here that will work regardless of how this
# python file was accessed (either through 'import', through 'python -m', or
# directly.
If the script is not at the top level of the package directory and you need to import a module below the top level, then the os.path.dirname has to be repeated until the parent_dir is the directory containing the top level.
Inspired by extraneon’s and taherh’s answers here is some code that runs up the file tree until it runs out of __init__.py files to build the full package name. This is definitely hacky, but does seem to work regardless of the depth of the file in your directory tree. It seems absolute imports are heavily encouraged.
import os, sys
if __name__ == "__main__" and __package__ is None:
d,f = os.path.split(os.path.abspath(__file__))
f = os.path.splitext(f)[0]
__package__ = [f] #__package__ will be a reversed list of package name parts
while os.path.exists(os.path.join(d,'__init__.py')): #go up until we run out of __init__.py files
d,name = os.path.split(d) #pull of a lowest level directory name
__package__.append(name) #add it to the package parts list
__package__ = ".".join(reversed(__package__)) #create the full package name
mod = __import__(__package__) #this assumes the top level package is in your $PYTHONPATH
sys.modules[__package__] = mod #add to modules
This is a minimal setup based on most of the other answers, tested on python 2.7 with a package layout like so. It also has the advantage that you can call the runme.py script from anywhere and it seems like it’s doing the right thing – I haven’t yet tested it in a more complex setup, so caveat emptor… etc.
This is basically Brad’s answer above with the insert into sys.path others have described.
packagetest/
__init__.py # Empty
mylib/
__init__.py # Empty
utils.py # def times2(x): return x*2
scripts/
__init__.py # Empty
runme.py # See below (executable)
runme.py looks like this:
#!/usr/bin/env python
if __name__ == '__main__' and __package__ is None:
from os import sys, path
d = path.dirname(path.abspath(__file__))
__package__ = []
while path.exists(path.join(d, '__init__.py')):
d, name = path.split(d)
__package__.append(name)
__package__ = ".".join(reversed(__package__))
sys.path.insert(1, d)
mod = __import__(__package__)
sys.modules[__package__] = mod
from ..mylib.utils import times2
print times2(4)
I have a “canonical file structure” like that (I’m giving sensible names to ease the reading):
mainpack/
__main__.py
__init__.py
- helpers/
__init__.py
path.py
- network/
__init__.py
clientlib.py
server.py
- gui/
__init__.py
mainwindow.py
controllers.py
In this structure, for example modules contained in each package may want to access the helpers utilities through relative imports in something like:
# network/clientlib.py
from ..helpers.path import create_dir
The program is runned “as a script” using the __main__.py file in this way:
python mainpack/
Trying to follow the PEP 366 I’ve put in __main__.py these lines:
___package___ = "mainpack"
from .network.clientlib import helloclient
But when running:
$ python mainpack
Traceback (most recent call last):
File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "path/mainpack/__main__.py", line 2, in <module>
from .network.clientlib import helloclient
SystemError: Parent module 'mainpack' not loaded, cannot perform relative import
What’s wrong? What is the correct way to handle and effectively use relative imports?
I’ve tried also to add the current directory to the PYTHONPATH, nothing changes.
The loading code seems to be something like this:
try:
return sys.modules[pkgname]
except KeyError:
if level < 1:
warn("Parent module '%s' not found while handling "
"absolute import" % pkgname, RuntimeWarning, 1)
return None
else:
raise SystemError, ("Parent module '%s' not loaded, cannot "
"perform relative import" % pkgname)
which makes me think that maybe your module is not on sys.path. If you start Python (normally) and just type “import mainpack” on the prompt, what does it do? It should be able to find it.
I have tried it myself and got the same error. After reading a bit I found the following solution:
# foo/__main__.py
import sys
mod = __import__('foo')
sys.modules["foo"]=mod
__package__='foo'
from .bar import hello
hello()
It seems a bit hackish to me but it does work. The trick seems to be making sure package foo is loaded so the import can be relative.
The “boilerplate” given in PEP 366 seems incomplete. Although it sets the __package__ variable, it doesn’t actually import the package, which is also needed to allow relative imports to work. extraneon‘s solution is on the right track.
Note that it is not enough to simply have the directory containing the module in sys.path, the corresponding package needs to be explicitly imported. The following seems like a better boilerplate than what was given in PEP 366 for ensuring that a python module can be executed regardless of how it is invoked (through a regular import, or with python -m, or with python, from any location):
# boilerplate to allow running as script directly
if __name__ == "__main__" and __package__ is None:
import sys, os
# The following assumes the script is in the top level of the package
# directory. We use dirname() to help get the parent directory to add to
# sys.path, so that we can import the current package. This is necessary
# since when invoked directly, the 'current' package is not automatically
# imported.
parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(1, parent_dir)
import mypackage
__package__ = str("mypackage")
del sys, os
# now you can use relative imports here that will work regardless of how this
# python file was accessed (either through 'import', through 'python -m', or
# directly.
If the script is not at the top level of the package directory and you need to import a module below the top level, then the os.path.dirname has to be repeated until the parent_dir is the directory containing the top level.
Inspired by extraneon’s and taherh’s answers here is some code that runs up the file tree until it runs out of __init__.py files to build the full package name. This is definitely hacky, but does seem to work regardless of the depth of the file in your directory tree. It seems absolute imports are heavily encouraged.
import os, sys
if __name__ == "__main__" and __package__ is None:
d,f = os.path.split(os.path.abspath(__file__))
f = os.path.splitext(f)[0]
__package__ = [f] #__package__ will be a reversed list of package name parts
while os.path.exists(os.path.join(d,'__init__.py')): #go up until we run out of __init__.py files
d,name = os.path.split(d) #pull of a lowest level directory name
__package__.append(name) #add it to the package parts list
__package__ = ".".join(reversed(__package__)) #create the full package name
mod = __import__(__package__) #this assumes the top level package is in your $PYTHONPATH
sys.modules[__package__] = mod #add to modules
This is a minimal setup based on most of the other answers, tested on python 2.7 with a package layout like so. It also has the advantage that you can call the runme.py script from anywhere and it seems like it’s doing the right thing – I haven’t yet tested it in a more complex setup, so caveat emptor… etc.
This is basically Brad’s answer above with the insert into sys.path others have described.
packagetest/
__init__.py # Empty
mylib/
__init__.py # Empty
utils.py # def times2(x): return x*2
scripts/
__init__.py # Empty
runme.py # See below (executable)
runme.py looks like this:
#!/usr/bin/env python
if __name__ == '__main__' and __package__ is None:
from os import sys, path
d = path.dirname(path.abspath(__file__))
__package__ = []
while path.exists(path.join(d, '__init__.py')):
d, name = path.split(d)
__package__.append(name)
__package__ = ".".join(reversed(__package__))
sys.path.insert(1, d)
mod = __import__(__package__)
sys.modules[__package__] = mod
from ..mylib.utils import times2
print times2(4)