What is __path__ useful for?
Question:
I had never noticed the __path__ attribute that gets defined on some of my packages before today. According to the documentation:
Packages support one more special
attribute, __path__. This is
initialized to be a list containing
the name of the directory holding the
package’s __init__.py before the code
in that file is executed. This
variable can be modified; doing so
affects future searches for modules
and subpackages contained in the
package.
While this feature is not often
needed, it can be used to extend the
set of modules found in a package.
Could somebody explain to me what exactly this means and why I would ever want to use it?
Answers:
If you change __path__, you can force the interpreter to look in a different directory for modules belonging to that package.
This would allow you to, e.g., load different versions of the same module based on runtime conditions. You might do this if you wanted to use different implementations of the same functionality on different platforms.
In addition to selecting different versions of a module based on runtime conditions as Syntactic says, this functionality also would allow you to break up your package into multiple pieces / downloads / installs while maintaining the appearance of a single logical package.
Consider the following.
- I have two packages,
mypkg and _mypkg_foo.
_mypkg_foo contains optional module to mypkg, foo.py. - as downloaded and installed,
mypkg doesn’t contain a foo.py.
mypkg‘s __init__.py can do something like so:
try:
import _mypkg_foo
__path__.append(os.path.abspath(os.path.dirname(_mypkg_foo.__file__)))
import mypkg.foo
except ImportError:
pass
If someone has installed the package _mypkg_foo, then mypkg.foo is available to them. If they haven’t, it doesn’t exist.
This is usually used with pkgutil to let a package be laid out across the disk. E.g., zope.interface and zope.schema are separate distributions (zope is a “namespace package”). You might have zope.interface installed in /usr/lib/python2.6/site-packages/zope/interface/, while you are using zope.schema more locally in /home/me/src/myproject/lib/python2.6/site-packages/zope/schema.
If you put pkgutil.extend_path(__path__, __name__) in /usr/lib/python2.6/site-packages/zope/__init__.py then both zope.interface and zope.schema will be importable because pkgutil will have change __path__ to ['/usr/lib/python2.6/site-packages/zope', '/home/me/src/myproject/lib/python2.6/site-packages/zope'].
pkg_resources.declare_namespace (part of Setuptools) is like pkgutil.extend_path but is more aware of zips on the path.
Manually changing __path__ is uncommon and probably not necessary, though it is useful to look at the variable when debugging import problems with namespace packages.
You can also use __path__ for monkeypatching, e.g., I have monkeypatched distutils at times by creating a file distutils/__init__.py that is early on sys.path:
import os
stdlib_dir = os.path.dirname(os.__file__)
real_distutils_path = os.path.join(stdlib_dir, 'distutils')
__path__.append(real_distutils_path)
execfile(os.path.join(real_distutils_path, '__init__.py'))
# and then apply some monkeypatching here...
A particular situation I’ve come across is when a package becomes large enough that I want to split parts of it into subdirectories without having to change any code that references it.
For example, I have a package called views that was collecting a number of supporting utility functions that were getting muddled with the main top-level purpose of the package. I was able to move these supporting functions into a subdirectory utils and add the following line to the __init__.py for the views package:
__path__.append(os.path.join(os.path.dirname(__file__), "utils"))
With this change too views/__init_.py, I could run the rest of the software with the new file structure without any further changes to the files.
(I tried to do something similar with import statements in the views/__init__.py file, but the sub-package modules were still not visible through an import of the view package – I’m not entirely sure if I’m missing something there; comments on that welcome!)
(This response based on Python 2.7 installation)
I had never noticed the __path__ attribute that gets defined on some of my packages before today. According to the documentation:
Packages support one more special
attribute,__path__. This is
initialized to be a list containing
the name of the directory holding the
package’s__init__.pybefore the code
in that file is executed. This
variable can be modified; doing so
affects future searches for modules
and subpackages contained in the
package.While this feature is not often
needed, it can be used to extend the
set of modules found in a package.
Could somebody explain to me what exactly this means and why I would ever want to use it?
If you change __path__, you can force the interpreter to look in a different directory for modules belonging to that package.
This would allow you to, e.g., load different versions of the same module based on runtime conditions. You might do this if you wanted to use different implementations of the same functionality on different platforms.
In addition to selecting different versions of a module based on runtime conditions as Syntactic says, this functionality also would allow you to break up your package into multiple pieces / downloads / installs while maintaining the appearance of a single logical package.
Consider the following.
- I have two packages,
mypkgand_mypkg_foo. _mypkg_foocontains optional module tomypkg,foo.py.- as downloaded and installed,
mypkgdoesn’t contain afoo.py.
mypkg‘s __init__.py can do something like so:
try:
import _mypkg_foo
__path__.append(os.path.abspath(os.path.dirname(_mypkg_foo.__file__)))
import mypkg.foo
except ImportError:
pass
If someone has installed the package _mypkg_foo, then mypkg.foo is available to them. If they haven’t, it doesn’t exist.
This is usually used with pkgutil to let a package be laid out across the disk. E.g., zope.interface and zope.schema are separate distributions (zope is a “namespace package”). You might have zope.interface installed in /usr/lib/python2.6/site-packages/zope/interface/, while you are using zope.schema more locally in /home/me/src/myproject/lib/python2.6/site-packages/zope/schema.
If you put pkgutil.extend_path(__path__, __name__) in /usr/lib/python2.6/site-packages/zope/__init__.py then both zope.interface and zope.schema will be importable because pkgutil will have change __path__ to ['/usr/lib/python2.6/site-packages/zope', '/home/me/src/myproject/lib/python2.6/site-packages/zope'].
pkg_resources.declare_namespace (part of Setuptools) is like pkgutil.extend_path but is more aware of zips on the path.
Manually changing __path__ is uncommon and probably not necessary, though it is useful to look at the variable when debugging import problems with namespace packages.
You can also use __path__ for monkeypatching, e.g., I have monkeypatched distutils at times by creating a file distutils/__init__.py that is early on sys.path:
import os
stdlib_dir = os.path.dirname(os.__file__)
real_distutils_path = os.path.join(stdlib_dir, 'distutils')
__path__.append(real_distutils_path)
execfile(os.path.join(real_distutils_path, '__init__.py'))
# and then apply some monkeypatching here...
A particular situation I’ve come across is when a package becomes large enough that I want to split parts of it into subdirectories without having to change any code that references it.
For example, I have a package called views that was collecting a number of supporting utility functions that were getting muddled with the main top-level purpose of the package. I was able to move these supporting functions into a subdirectory utils and add the following line to the __init__.py for the views package:
__path__.append(os.path.join(os.path.dirname(__file__), "utils"))
With this change too views/__init_.py, I could run the rest of the software with the new file structure without any further changes to the files.
(I tried to do something similar with import statements in the views/__init__.py file, but the sub-package modules were still not visible through an import of the view package – I’m not entirely sure if I’m missing something there; comments on that welcome!)
(This response based on Python 2.7 installation)