Expand Python Search Path to Other Source
Question:
I have just joined a project with a rather large existing code base. We develop in linux and do not use and IDE. We run through the command line. I’m trying to figure out how to get python to search for the right path when I run project modules. For instance, when I run something like:
python someprojectfile.py
I get
ImportError: no module named core.'somemodule'
I get this for all of my imports to I assume it’s an issue with the path.
How do I get Python to search ~/codez/project/ and all the files and folders for *.py files during import statements?
Answers:
There are a few possible ways to do this:
- Set the environment variable
PYTHONPATH to a colon-separated list of directories to search for imported modules.
- In your program, use
sys.path.append('/path/to/search') to add the names of directories you want Python to search for imported modules. sys.path is just the list of directories Python searches every time it gets asked to import a module, and you can alter it as needed (although I wouldn’t recommend removing any of the standard directories!). Any directories you put in the environment variable PYTHONPATH will be inserted into sys.path when Python starts up.
- Use
site.addsitedir to add a directory to sys.path. The difference between this and just plain appending is that when you use addsitedir, it also looks for .pth files within that directory and uses them to possibly add additional directories to sys.path based on the contents of the files. See the documentation for more detail.
Which one of these you want to use depends on your situation. Remember that when you distribute your project to other users, they typically install it in such a manner that the Python code files will be automatically detected by Python’s importer (i.e. packages are usually installed in the site-packages directory), so if you mess with sys.path in your code, that may be unnecessary and might even have adverse effects when that code runs on another computer. For development, I would venture a guess that setting PYTHONPATH is usually the best way to go.
However, when you’re using something that just runs on your own computer (or when you have nonstandard setups, e.g. sometimes in web app frameworks), it’s not entirely uncommon to do something like
import sys
from os.path import dirname
sys.path.append(dirname(__file__))
You should also read about python packages here: http://docs.python.org/tutorial/modules.html.
From your example, I would guess that you really have a package at ~/codez/project. The file __init__.py in a python directory maps a directory into a namespace. If your subdirectories all have an __init__.py file, then you only need to add the base directory to your PYTHONPATH. For example:
PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/project
In addition to testing your PYTHONPATH environment variable, as David explains, you can test it in python like this:
$ python
>>> import project # should work if PYTHONPATH set
>>> import sys
>>> for line in sys.path: print line # print current python path
…
I read this question looking for an answer, and didn’t like any of them.
So I wrote a quick and dirty solution. Just put this somewhere on your sys.path, and it’ll add any directory under folder (from the current working directory), or under abspath:
#using.py
import sys, os.path
def all_from(folder='', abspath=None):
"""add all dirs under `folder` to sys.path if any .py files are found.
Use an abspath if you'd rather do it that way.
Uses the current working directory as the location of using.py.
Keep in mind that os.walk goes *all the way* down the directory tree.
With that, try not to use this on something too close to '/'
"""
add = set(sys.path)
if abspath is None:
cwd = os.path.abspath(os.path.curdir)
abspath = os.path.join(cwd, folder)
for root, dirs, files in os.walk(abspath):
for f in files:
if f[-3:] in '.py':
add.add(root)
break
for i in add: sys.path.append(i)
>>> import using, sys, pprint
>>> using.all_from('py') #if in ~, /home/user/py/
>>> pprint.pprint(sys.path)
[
#that was easy
]
And I like it because I can have a folder for some random tools and not have them be a part of packages or anything, and still get access to some (or all) of them in a couple lines of code.
The easiest way I find is to create a file any_name.pth and put it in your folder Libsite-packages. You should find that folder wherever python is installed.
In that file, put a list of directories where you want to keep modules for importing. For instance, make a line in that file like this:
C:Usersexample...example
You will be able to tell it works by running this in python:
import sys
for line in sys.path:
print line
You will see your directory printed out, amongst others from where you can also import. Now you can import a mymodule.py file that sits in that directory as easily as:
import mymodule
This will not import subfolders. For that you could imagine creating a python script to create a .pth file containing all sub folders of a folder you define. Have it run at startup perhaps.
I know this thread is a bit old, but it took me some time to get to the heart of this, so I wanted to share.
In my project, I had the main script in a parent directory, and, to differentiate the modules, I put all the supporting modules in a sub-folder called “modules”. In my main script, I import these modules like this (for a module called report.py):
from modules.report import report, reportError
If I call my main script, this works. HOWEVER, I wanted to test each module by including a main() in each, and calling each directly, as:
python modules/report.py
Now Python complains that it can’t find “a module called modules”. The key here is that, by default, Python includes the folder of the script in its search path, BUT NOT THE CWD. So what this error says, really, is “I can’t find a modules subfolder”. The is because there is no “modules” subdirectory from the directory where the report.py module resides.
I find that the neatest solution to this is to append the CWD in Python search path by including this at the top:
import sys
sys.path.append(".")
Now Python searches the CWD (current directory), finds the “modules” sub-folder, and all is well.
New option for old question.
Installing fail2ban package on Debian, looks like it’s hardcoded to install on /usr/lib/python3/dist-packages/fail2ban a path not on python3 sys.path.
> python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Jun 25 2019, 18:51:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
>>>
so, instead of just copying, I (bash) linked the library to newer versions.
Future updates to the original app, will also be automatically applied to the linked versions.
if [ -d /usr/lib/python3/dist-packages/fail2ban ]
then
for d in /usr/lib/python3.*
do
[ -d ${d}/fail2ban ] ||
ln -vs /usr/lib/python3/dist-packages/fail2ban ${d}/
done
fi
the first task is to find out where the current environment gets its standard libraries from. And that’s where I would store my OOP modules.
Import numpy
numpy.__file__
You get a catalog on your environment. For another computer, it is a good idea to create a separate directory for OOP modules.
Import sys
sys.path.append('F:/Python/My_libra/')
I have just joined a project with a rather large existing code base. We develop in linux and do not use and IDE. We run through the command line. I’m trying to figure out how to get python to search for the right path when I run project modules. For instance, when I run something like:
python someprojectfile.py
I get
ImportError: no module named core.'somemodule'
I get this for all of my imports to I assume it’s an issue with the path.
How do I get Python to search ~/codez/project/ and all the files and folders for *.py files during import statements?
There are a few possible ways to do this:
- Set the environment variable
PYTHONPATHto a colon-separated list of directories to search for imported modules. - In your program, use
sys.path.append('/path/to/search')to add the names of directories you want Python to search for imported modules.sys.pathis just the list of directories Python searches every time it gets asked to import a module, and you can alter it as needed (although I wouldn’t recommend removing any of the standard directories!). Any directories you put in the environment variablePYTHONPATHwill be inserted intosys.pathwhen Python starts up. - Use
site.addsitedirto add a directory tosys.path. The difference between this and just plain appending is that when you useaddsitedir, it also looks for.pthfiles within that directory and uses them to possibly add additional directories tosys.pathbased on the contents of the files. See the documentation for more detail.
Which one of these you want to use depends on your situation. Remember that when you distribute your project to other users, they typically install it in such a manner that the Python code files will be automatically detected by Python’s importer (i.e. packages are usually installed in the site-packages directory), so if you mess with sys.path in your code, that may be unnecessary and might even have adverse effects when that code runs on another computer. For development, I would venture a guess that setting PYTHONPATH is usually the best way to go.
However, when you’re using something that just runs on your own computer (or when you have nonstandard setups, e.g. sometimes in web app frameworks), it’s not entirely uncommon to do something like
import sys
from os.path import dirname
sys.path.append(dirname(__file__))
You should also read about python packages here: http://docs.python.org/tutorial/modules.html.
From your example, I would guess that you really have a package at ~/codez/project. The file __init__.py in a python directory maps a directory into a namespace. If your subdirectories all have an __init__.py file, then you only need to add the base directory to your PYTHONPATH. For example:
PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/project
In addition to testing your PYTHONPATH environment variable, as David explains, you can test it in python like this:
$ python
>>> import project # should work if PYTHONPATH set
>>> import sys
>>> for line in sys.path: print line # print current python path
…
I read this question looking for an answer, and didn’t like any of them.
So I wrote a quick and dirty solution. Just put this somewhere on your sys.path, and it’ll add any directory under folder (from the current working directory), or under abspath:
#using.py
import sys, os.path
def all_from(folder='', abspath=None):
"""add all dirs under `folder` to sys.path if any .py files are found.
Use an abspath if you'd rather do it that way.
Uses the current working directory as the location of using.py.
Keep in mind that os.walk goes *all the way* down the directory tree.
With that, try not to use this on something too close to '/'
"""
add = set(sys.path)
if abspath is None:
cwd = os.path.abspath(os.path.curdir)
abspath = os.path.join(cwd, folder)
for root, dirs, files in os.walk(abspath):
for f in files:
if f[-3:] in '.py':
add.add(root)
break
for i in add: sys.path.append(i)
>>> import using, sys, pprint
>>> using.all_from('py') #if in ~, /home/user/py/
>>> pprint.pprint(sys.path)
[
#that was easy
]
And I like it because I can have a folder for some random tools and not have them be a part of packages or anything, and still get access to some (or all) of them in a couple lines of code.
The easiest way I find is to create a file any_name.pth and put it in your folder Libsite-packages. You should find that folder wherever python is installed.
In that file, put a list of directories where you want to keep modules for importing. For instance, make a line in that file like this:
C:Usersexample...example
You will be able to tell it works by running this in python:
import sys
for line in sys.path:
print line
You will see your directory printed out, amongst others from where you can also import. Now you can import a mymodule.py file that sits in that directory as easily as:
import mymodule
This will not import subfolders. For that you could imagine creating a python script to create a .pth file containing all sub folders of a folder you define. Have it run at startup perhaps.
I know this thread is a bit old, but it took me some time to get to the heart of this, so I wanted to share.
In my project, I had the main script in a parent directory, and, to differentiate the modules, I put all the supporting modules in a sub-folder called “modules”. In my main script, I import these modules like this (for a module called report.py):
from modules.report import report, reportError
If I call my main script, this works. HOWEVER, I wanted to test each module by including a main() in each, and calling each directly, as:
python modules/report.py
Now Python complains that it can’t find “a module called modules”. The key here is that, by default, Python includes the folder of the script in its search path, BUT NOT THE CWD. So what this error says, really, is “I can’t find a modules subfolder”. The is because there is no “modules” subdirectory from the directory where the report.py module resides.
I find that the neatest solution to this is to append the CWD in Python search path by including this at the top:
import sys
sys.path.append(".")
Now Python searches the CWD (current directory), finds the “modules” sub-folder, and all is well.
New option for old question.
Installing fail2ban package on Debian, looks like it’s hardcoded to install on /usr/lib/python3/dist-packages/fail2ban a path not on python3 sys.path.
> python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Jun 25 2019, 18:51:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
>>>
so, instead of just copying, I (bash) linked the library to newer versions.
Future updates to the original app, will also be automatically applied to the linked versions.
if [ -d /usr/lib/python3/dist-packages/fail2ban ]
then
for d in /usr/lib/python3.*
do
[ -d ${d}/fail2ban ] ||
ln -vs /usr/lib/python3/dist-packages/fail2ban ${d}/
done
fi
the first task is to find out where the current environment gets its standard libraries from. And that’s where I would store my OOP modules.
Import numpy
numpy.__file__
You get a catalog on your environment. For another computer, it is a good idea to create a separate directory for OOP modules.
Import sys
sys.path.append('F:/Python/My_libra/')