What sets up sys.path with Python, and when?
Question:
When I run
import sys
print sys.path
on my Mac (Mac OS X 10.6.5, Python 2.6.1), I get the following results.
/Library/Python/2.6/site-packages/ply-3.3-py2.6.egg
...
/Library/Python/2.6/site-packages/ipython-0.10.1-py2.6.egg
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Python/2.6/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode
They are grouped into 5 categories.
- /Library/Python/2.6/site-packages/*.egg
- /Library/Python/2.6/site-packages
- Frameworks/Python.framework/Versions/2.6/lib/python2.6
- Frameworks/Python.framework/Versions/2.6/Extras/lib/python
- PATH from PYTHONPATH environment variable.
And I can add more paths using the code
sys.path.insert(0, MORE_PATH)
- What routines sets up those paths, and when?
- Are some of the paths are built in python source code?
- Is it possible that the paths inserted with ‘sys.path.insert’ are ignored? I’m curious about this, as with mod_wsgi, I found the paths are not found with ‘sys.path.insert’. I asked another post for this question.
ADDED
Based on Michael’s answer, I looked into site.py, and I got the following code.
def addsitepackages(known_paths):
"""Add site-packages (and possibly site-python) to sys.path"""
sitedirs = []
seen = []
for prefix in PREFIXES:
if not prefix or prefix in seen:
continue
seen.append(prefix)
if sys.platform in ('os2emx', 'riscos'):
sitedirs.append(os.path.join(prefix, "Lib", "site-packages"))
elif sys.platform == 'darwin' and prefix == sys.prefix:
sitedirs.append(os.path.join("/Library/Python", sys.version[:3], "site-packages"))
I also think that the directory name that has site.py (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6 for my Mac) should be built into Python source code.
Answers:
From Learning Python:
sys.path is the module search path.
Python configures it at program
startup, automatically merging the
home directory of the top-level file
(or an empty string to designate the
current working directory), any
PYTHONPATH directories, the contents
of any .pth file paths you’ve
created, and the standard library
directories. The result is a list of
directory name strings that Python
searches on each import of a new file.
Most of the stuff is set up in Python’s site.py which is automatically imported when starting the interpreter (unless you start it with the -S option). Few paths are set up in the interpreter itself during initialization (you can find out which by starting python with -S).
Additionally, some frameworks (like Django I think) modify sys.path upon startup to meet their requirements.
The site module has a pretty good documentation, a commented source code and prints out some information if you run it via python -m site.
Path has these parts:
- OS paths that have your system libraries
- current directory python started from
- environmental variable
$PYTHONPATH
- you can add paths at runtime.
Paths are not ignored. But, they may not be found and that will not raise an error.
sys.path should only be added too, not subtracted from. Django would not remove paths.
site.py is indeed the answers. I wanted to remove any dependencies on the old Python that is installed by default on my mac. This works pretty good, as ‘site.py’ is called each time the python interpreter is started.
For Mac, I manually added the following line at the end of main() in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/site.py:
sys.path = filter (lambda a: not a.startswith('/System'), sys.path)
Adding to the accepted answer, and addressing the comments that say a module shouldn’t remove entries from sys.path:
This is broadly true but there are circumstances where you might want to modify sys.path by removing entries. For instance – and this is Mac-specific; *nix/Windows corollaries may exist – if you create a customised Python.framework for inclusion in your own project you may want to ignore the default sys.path entries that point at the system Python.framework.
You have a couple of options:
-
Hack the site.py, as @damirv indicates, or
-
Add your own sitecustomize module (or package) to the custom framework that achieves the same end result. As indicated in the site.py comments (for 2.7.6, anyway):
After these path manipulations, an attempt is made to import a module
named sitecustomize, which can perform arbitrary additional
site-specific customizations. If this import fails with an
ImportError exception, it is silently ignored.
Also note: if the PYTHONHOME env var is set, standard libraries will be loaded from this path instead of the default, as documented.
This is not a direct answer to the question, but something I just discovered that was causing the wrong standard libraries to be loaded, and my searches lead me here along the way.
You are using system python /usr/bin/python.
sys.path is set from system files at python startup.
Do not touch those files, in particular site.py, because this may perturb the system.
However, you can change sys.path within python, in particular, at startup :
in ~/.bashrc or ~/.zshrc:
export PYTHONSTARTUP=~/.pythonrc
in ~/.pythonrc:
write your changes to sys.path.
Those changes will be only for you in interactive shells.
For hacking at little risk for the system, install you own and more recent python version.
When I run
import sys
print sys.path
on my Mac (Mac OS X 10.6.5, Python 2.6.1), I get the following results.
/Library/Python/2.6/site-packages/ply-3.3-py2.6.egg ... /Library/Python/2.6/site-packages/ipython-0.10.1-py2.6.egg /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6 /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload /Library/Python/2.6/site-packages /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode
They are grouped into 5 categories.
- /Library/Python/2.6/site-packages/*.egg
- /Library/Python/2.6/site-packages
- Frameworks/Python.framework/Versions/2.6/lib/python2.6
- Frameworks/Python.framework/Versions/2.6/Extras/lib/python
- PATH from PYTHONPATH environment variable.
And I can add more paths using the code
sys.path.insert(0, MORE_PATH)
- What routines sets up those paths, and when?
- Are some of the paths are built in python source code?
- Is it possible that the paths inserted with ‘sys.path.insert’ are ignored? I’m curious about this, as with mod_wsgi, I found the paths are not found with ‘sys.path.insert’. I asked another post for this question.
ADDED
Based on Michael’s answer, I looked into site.py, and I got the following code.
def addsitepackages(known_paths):
"""Add site-packages (and possibly site-python) to sys.path"""
sitedirs = []
seen = []
for prefix in PREFIXES:
if not prefix or prefix in seen:
continue
seen.append(prefix)
if sys.platform in ('os2emx', 'riscos'):
sitedirs.append(os.path.join(prefix, "Lib", "site-packages"))
elif sys.platform == 'darwin' and prefix == sys.prefix:
sitedirs.append(os.path.join("/Library/Python", sys.version[:3], "site-packages"))
I also think that the directory name that has site.py (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6 for my Mac) should be built into Python source code.
From Learning Python:
sys.path is the module search path.
Python configures it at program
startup, automatically merging the
home directory of the top-level file
(or an empty string to designate the
current working directory), any
PYTHONPATH directories, the contents
of any .pth file paths you’ve
created, and the standard library
directories. The result is a list of
directory name strings that Python
searches on each import of a new file.
Most of the stuff is set up in Python’s site.py which is automatically imported when starting the interpreter (unless you start it with the -S option). Few paths are set up in the interpreter itself during initialization (you can find out which by starting python with -S).
Additionally, some frameworks (like Django I think) modify sys.path upon startup to meet their requirements.
The site module has a pretty good documentation, a commented source code and prints out some information if you run it via python -m site.
Path has these parts:
- OS paths that have your system libraries
- current directory python started from
- environmental variable
$PYTHONPATH - you can add paths at runtime.
Paths are not ignored. But, they may not be found and that will not raise an error.
sys.path should only be added too, not subtracted from. Django would not remove paths.
site.py is indeed the answers. I wanted to remove any dependencies on the old Python that is installed by default on my mac. This works pretty good, as ‘site.py’ is called each time the python interpreter is started.
For Mac, I manually added the following line at the end of main() in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/site.py:
sys.path = filter (lambda a: not a.startswith('/System'), sys.path)
Adding to the accepted answer, and addressing the comments that say a module shouldn’t remove entries from sys.path:
This is broadly true but there are circumstances where you might want to modify sys.path by removing entries. For instance – and this is Mac-specific; *nix/Windows corollaries may exist – if you create a customised Python.framework for inclusion in your own project you may want to ignore the default sys.path entries that point at the system Python.framework.
You have a couple of options:
-
Hack the
site.py, as @damirv indicates, or -
Add your own
sitecustomizemodule (or package) to the custom framework that achieves the same end result. As indicated in thesite.pycomments (for 2.7.6, anyway):After these path manipulations, an attempt is made to import a module
named sitecustomize, which can perform arbitrary additional
site-specific customizations. If this import fails with an
ImportError exception, it is silently ignored.
Also note: if the PYTHONHOME env var is set, standard libraries will be loaded from this path instead of the default, as documented.
This is not a direct answer to the question, but something I just discovered that was causing the wrong standard libraries to be loaded, and my searches lead me here along the way.
You are using system python /usr/bin/python.
sys.path is set from system files at python startup.
Do not touch those files, in particular site.py, because this may perturb the system.
However, you can change sys.path within python, in particular, at startup :
in ~/.bashrc or ~/.zshrc:
export PYTHONSTARTUP=~/.pythonrc
in ~/.pythonrc:
write your changes to sys.path.
Those changes will be only for you in interactive shells.
For hacking at little risk for the system, install you own and more recent python version.