Import local function from a module housed in another directory with relative imports in Jupyter Notebook using Python 3
Question:
I have a directory structure similar to the following
meta_project
project1
__init__.py
lib
module.py
__init__.py
notebook_folder
notebook.jpynb
When working in notebook.jpynb if I try to use a relative import to access a function function() in module.py with:
from ..project1.lib.module import function
I get the following error:
SystemError Traceback (most recent call last)
<ipython-input-7-6393744d93ab> in <module>()
----> 1 from ..project1.lib.module import function
SystemError: Parent module '' not loaded, cannot perform relative import
Is there any way to get this to work using relative imports?
Note, the notebook server is instantiated at the level of the meta_project directory, so it should have access to the information in those files.
Note, also, that at least as originally intended project1 wasn’t thought of as a module and therefore does not have an __init__.py file, it was just meant as a file-system directory. If the solution to the problem requires treating it as a module and including an __init__.py file (even a blank one) that is fine, but doing so is not enough to solve the problem.
I share this directory between machines and relative imports allow me to use the same code everywhere, & I often use notebooks for quick prototyping, so suggestions that involve hacking together absolute paths are unlikely to be helpful.
Edit: This is unlike Relative imports in Python 3, which talks about relative imports in Python 3 in general and – in particular – running a script from within a package directory. This has to do with working within a jupyter notebook trying to call a function in a local module in another directory which has both different general and particular aspects.
Answers:
I had almost the same example as you in this notebook where I wanted to illustrate the usage of an adjacent module’s function in a DRY manner.
My solution was to tell Python of that additional module import path by adding a snippet like this one to the notebook:
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
This allows you to import the desired function from the module hierarchy:
from project1.lib.module import function
# use the function normally
function(...)
Note that it is necessary to add empty __init__.py files to project1/ and lib/ folders if you don’t have them already.
Came here searching for best practices in abstracting code to submodules when working in Notebooks. I’m not sure that there is a best practice. I have been proposing this.
A project hierarchy as such:
├── ipynb
│ ├── 20170609-Examine_Database_Requirements.ipynb
│ └── 20170609-Initial_Database_Connection.ipynb
└── lib
├── __init__.py
└── postgres.py
And from 20170609-Initial_Database_Connection.ipynb:
In [1]: cd ..
In [2]: from lib.postgres import database_connection
This works because by default the Jupyter Notebook can parse the cd command. Note that this does not make use of Python Notebook magic. It simply works without prepending %bash.
Considering that 99 times out of a 100 I am working in Docker using one of the Project Jupyter Docker images, the following modification is idempotent
In [1]: cd /home/jovyan
In [2]: from lib.postgres import database_connection
So far, the accepted answer has worked best for me. However, my concern has always been that there is a likely scenario where I might refactor the notebooks directory into subdirectories, requiring to change the module_path in every notebook. I decided to add a python file within each notebook directory to import the required modules.
Thus, having the following project structure:
project
|__notebooks
|__explore
|__ notebook1.ipynb
|__ notebook2.ipynb
|__ project_path.py
|__ explain
|__notebook1.ipynb
|__project_path.py
|__lib
|__ __init__.py
|__ module.py
I added the file project_path.py in each notebook subdirectory (notebooks/explore and notebooks/explain). This file contains the code for relative imports (from @metakermit):
import sys
import os
module_path = os.path.abspath(os.path.join(os.pardir, os.pardir))
if module_path not in sys.path:
sys.path.append(module_path)
This way, I just need to do relative imports within the project_path.py file, and not in the notebooks. The notebooks files would then just need to import project_path before importing lib. For example in 0.0-notebook.ipynb:
import project_path
import lib
The caveat here is that reversing the imports would not work. THIS DOES NOT WORK:
import lib
import project_path
Thus care must be taken during imports.
Researching this topic myself and having read the answers I recommend using the path.py library since it provides a context manager for changing the current working directory.
You then have something like
import path
if path.Path('../lib').isdir():
with path.Path('..'):
import lib
Although, you might just omit the isdir statement.
Here I’ll add print statements to make it easy to follow what’s happening
import path
import pandas
print(path.Path.getcwd())
print(path.Path('../lib').isdir())
if path.Path('../lib').isdir():
with path.Path('..'):
print(path.Path.getcwd())
import lib
print('Success!')
print(path.Path.getcwd())
which outputs in this example (where lib is at /home/jovyan/shared/notebooks/by-team/data-vis/demos/lib):
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
/home/jovyan/shared/notebooks/by-team/data-vis/demos
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
Since the solution uses a context manager, you are guaranteed to go back to your previous working directory, no matter what state your kernel was in before the cell and no matter what exceptions are thrown by importing your library code.
I have just found this pretty solution:
import sys; sys.path.insert(0, '..') # add parent folder path where lib folder is
import lib.store_load # store_load is a file on my library folder
You just want some functions of that file
from lib.store_load import your_function_name
If python version >= 3.3 you do not need init.py file in the folder
Here’s my 2 cents:
import sys
# map the path where the module file is located. In my case it was the desktop
sys.path.append('/Users/John/Desktop')
# Either import the whole mapping module BUT then you have to use the .notation to map the classes like mapping.Shipping()
import mapping #mapping.py is the name of my module file
shipit = mapping.Shipment() #Shipment is the name of the class I need to use in the mapping module
# Or import the specific class from the mapping module
from mapping import Mapping
shipit = Shipment() #Now you don't have to use the .notation
I have found that python-dotenv helps solve this issue pretty effectively. Your project structure ends up changing slightly, but the code in your notebook is a bit simpler and consistent across notebooks.
For your project, do a little install.
pipenv install python-dotenv
Then, project changes to:
├── .env (this can be empty)
├── ipynb
│ ├── 20170609-Examine_Database_Requirements.ipynb
│ └── 20170609-Initial_Database_Connection.ipynb
└── lib
├── __init__.py
└── postgres.py
And finally, your import changes to:
import os
import sys
from dotenv import find_dotenv
sys.path.append(os.path.dirname(find_dotenv()))
A +1 for this package is that your notebooks can be several directories deep. python-dotenv will find the closest one in a parent directory and use it. A +2 for this approach is that jupyter will load environment variables from the .env file on startup. Double whammy.
All other answers here depends on adding code the the notebook(!)
In my opinion is bad practice to hardcode a specific path into the notebook code, or otherwise depend on the location, since this makes it really hard to refactor you code later on. Instead I would recommend you to add the root project folder to PYTHONPATH when starting up your Jupyter notebook server, either directly from the project folder like so
env PYTHONPATH=`pwd` jupyter notebook
or if you are starting it up from somewhere else, use the absolute path like so
env PYTHONPATH=/Users/foo/bar/project/ jupyter notebook
For those who don’t understand the solution as I didn’t, you have to go as deep to the directory as you need for your specific problem. Solution for this errorr:
No module named ‘your_folder’
My notebook that I work with is in:
C:UsersvojteprojectspredictaEDAapartmentsEDA.ipynb
I want to import this notebook:
C:Usersvojteprojectsfunctionsfunctions.ipynb
I had to modify the solution above by @metakermit to this solution:
import os
import sys
module_path = os.path.abspath(os.path.join('......'))
if module_path not in sys.path:
sys.path.append(module_path)
from functions import functions as f
You can check the sys.path for appended directory. In this example this line was appended:
C:\Users\vojte\projects
Improving @joshua-cook answer in using cd.., to make sure that you did not re-run the cell and messing your directory in using run all without restart, use this code instead:
if 'NOTEBOOK_INITIATED_FLAG' not in globals():
NOTEBOOK_INITIATED_FLAG = True
%cd ..
%pwd
NOTEBOOK_INITIATED_FLAG is used as a placeholder that mark that the kernel is already running, thus changing directory is not needed.
And this is the super convoluted boilerplate if you want to work with jupytext and run your .py file from parent folder:
import os
import sys
if 'NOTEBOOK_INITIATED_FLAG' not in globals():
NOTEBOOK_INITIATED_FLAG = True
try:
# not in notebook
module_path = os.path.join(os.path.dirname(__file__), os.pardir)
except:
# in notebook
module_path = os.path.abspath(os.path.join('..'))
%cd ..
%pwd
if module_path not in sys.path:
sys.path.append(module_path)
I got annoyed so I wrote a package.
pip install package_root and then you can import package_root # into your PYTHONPATH.
This works by looking through the call stack to determine which file imported package_root and then walking the up directories to find the first directory that is missing an __init__.py file.
So in practice, you’ll need to add __init__.py files to your notebook directories just like you would to get Python packages to work.
If you are working via the Jupyter extension in VS Code, I found that you can now set this path in your settings.json file.
{
"jupyter.notebookFileRoot": "/path/to/your/module/root"
}
Edit: Or, to set it at your workspace root more generically:
{
"jupyter.notebookFileRoot": "${workspaceFolder}"
}
I have a directory structure similar to the following
meta_project
project1
__init__.py
lib
module.py
__init__.py
notebook_folder
notebook.jpynb
When working in notebook.jpynb if I try to use a relative import to access a function function() in module.py with:
from ..project1.lib.module import function
I get the following error:
SystemError Traceback (most recent call last)
<ipython-input-7-6393744d93ab> in <module>()
----> 1 from ..project1.lib.module import function
SystemError: Parent module '' not loaded, cannot perform relative import
Is there any way to get this to work using relative imports?
Note, the notebook server is instantiated at the level of the meta_project directory, so it should have access to the information in those files.
Note, also, that at least as originally intended project1 wasn’t thought of as a module and therefore does not have an __init__.py file, it was just meant as a file-system directory. If the solution to the problem requires treating it as a module and including an __init__.py file (even a blank one) that is fine, but doing so is not enough to solve the problem.
I share this directory between machines and relative imports allow me to use the same code everywhere, & I often use notebooks for quick prototyping, so suggestions that involve hacking together absolute paths are unlikely to be helpful.
Edit: This is unlike Relative imports in Python 3, which talks about relative imports in Python 3 in general and – in particular – running a script from within a package directory. This has to do with working within a jupyter notebook trying to call a function in a local module in another directory which has both different general and particular aspects.
I had almost the same example as you in this notebook where I wanted to illustrate the usage of an adjacent module’s function in a DRY manner.
My solution was to tell Python of that additional module import path by adding a snippet like this one to the notebook:
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
This allows you to import the desired function from the module hierarchy:
from project1.lib.module import function
# use the function normally
function(...)
Note that it is necessary to add empty __init__.py files to project1/ and lib/ folders if you don’t have them already.
Came here searching for best practices in abstracting code to submodules when working in Notebooks. I’m not sure that there is a best practice. I have been proposing this.
A project hierarchy as such:
├── ipynb
│ ├── 20170609-Examine_Database_Requirements.ipynb
│ └── 20170609-Initial_Database_Connection.ipynb
└── lib
├── __init__.py
└── postgres.py
And from 20170609-Initial_Database_Connection.ipynb:
In [1]: cd ..
In [2]: from lib.postgres import database_connection
This works because by default the Jupyter Notebook can parse the cd command. Note that this does not make use of Python Notebook magic. It simply works without prepending %bash.
Considering that 99 times out of a 100 I am working in Docker using one of the Project Jupyter Docker images, the following modification is idempotent
In [1]: cd /home/jovyan
In [2]: from lib.postgres import database_connection
So far, the accepted answer has worked best for me. However, my concern has always been that there is a likely scenario where I might refactor the notebooks directory into subdirectories, requiring to change the module_path in every notebook. I decided to add a python file within each notebook directory to import the required modules.
Thus, having the following project structure:
project
|__notebooks
|__explore
|__ notebook1.ipynb
|__ notebook2.ipynb
|__ project_path.py
|__ explain
|__notebook1.ipynb
|__project_path.py
|__lib
|__ __init__.py
|__ module.py
I added the file project_path.py in each notebook subdirectory (notebooks/explore and notebooks/explain). This file contains the code for relative imports (from @metakermit):
import sys
import os
module_path = os.path.abspath(os.path.join(os.pardir, os.pardir))
if module_path not in sys.path:
sys.path.append(module_path)
This way, I just need to do relative imports within the project_path.py file, and not in the notebooks. The notebooks files would then just need to import project_path before importing lib. For example in 0.0-notebook.ipynb:
import project_path
import lib
The caveat here is that reversing the imports would not work. THIS DOES NOT WORK:
import lib
import project_path
Thus care must be taken during imports.
Researching this topic myself and having read the answers I recommend using the path.py library since it provides a context manager for changing the current working directory.
You then have something like
import path
if path.Path('../lib').isdir():
with path.Path('..'):
import lib
Although, you might just omit the isdir statement.
Here I’ll add print statements to make it easy to follow what’s happening
import path
import pandas
print(path.Path.getcwd())
print(path.Path('../lib').isdir())
if path.Path('../lib').isdir():
with path.Path('..'):
print(path.Path.getcwd())
import lib
print('Success!')
print(path.Path.getcwd())
which outputs in this example (where lib is at /home/jovyan/shared/notebooks/by-team/data-vis/demos/lib):
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
/home/jovyan/shared/notebooks/by-team/data-vis/demos
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
Since the solution uses a context manager, you are guaranteed to go back to your previous working directory, no matter what state your kernel was in before the cell and no matter what exceptions are thrown by importing your library code.
I have just found this pretty solution:
import sys; sys.path.insert(0, '..') # add parent folder path where lib folder is
import lib.store_load # store_load is a file on my library folder
You just want some functions of that file
from lib.store_load import your_function_name
If python version >= 3.3 you do not need init.py file in the folder
Here’s my 2 cents:
import sys
# map the path where the module file is located. In my case it was the desktop
sys.path.append('/Users/John/Desktop')
# Either import the whole mapping module BUT then you have to use the .notation to map the classes like mapping.Shipping()
import mapping #mapping.py is the name of my module file
shipit = mapping.Shipment() #Shipment is the name of the class I need to use in the mapping module
# Or import the specific class from the mapping module
from mapping import Mapping
shipit = Shipment() #Now you don't have to use the .notation
I have found that python-dotenv helps solve this issue pretty effectively. Your project structure ends up changing slightly, but the code in your notebook is a bit simpler and consistent across notebooks.
For your project, do a little install.
pipenv install python-dotenv
Then, project changes to:
├── .env (this can be empty)
├── ipynb
│ ├── 20170609-Examine_Database_Requirements.ipynb
│ └── 20170609-Initial_Database_Connection.ipynb
└── lib
├── __init__.py
└── postgres.py
And finally, your import changes to:
import os
import sys
from dotenv import find_dotenv
sys.path.append(os.path.dirname(find_dotenv()))
A +1 for this package is that your notebooks can be several directories deep. python-dotenv will find the closest one in a parent directory and use it. A +2 for this approach is that jupyter will load environment variables from the .env file on startup. Double whammy.
All other answers here depends on adding code the the notebook(!)
In my opinion is bad practice to hardcode a specific path into the notebook code, or otherwise depend on the location, since this makes it really hard to refactor you code later on. Instead I would recommend you to add the root project folder to PYTHONPATH when starting up your Jupyter notebook server, either directly from the project folder like so
env PYTHONPATH=`pwd` jupyter notebook
or if you are starting it up from somewhere else, use the absolute path like so
env PYTHONPATH=/Users/foo/bar/project/ jupyter notebook
For those who don’t understand the solution as I didn’t, you have to go as deep to the directory as you need for your specific problem. Solution for this errorr:
No module named ‘your_folder’
My notebook that I work with is in:
C:UsersvojteprojectspredictaEDAapartmentsEDA.ipynb
I want to import this notebook:
C:Usersvojteprojectsfunctionsfunctions.ipynb
I had to modify the solution above by @metakermit to this solution:
import os
import sys
module_path = os.path.abspath(os.path.join('......'))
if module_path not in sys.path:
sys.path.append(module_path)
from functions import functions as f
You can check the sys.path for appended directory. In this example this line was appended:
C:\Users\vojte\projects
Improving @joshua-cook answer in using cd.., to make sure that you did not re-run the cell and messing your directory in using run all without restart, use this code instead:
if 'NOTEBOOK_INITIATED_FLAG' not in globals():
NOTEBOOK_INITIATED_FLAG = True
%cd ..
%pwd
NOTEBOOK_INITIATED_FLAG is used as a placeholder that mark that the kernel is already running, thus changing directory is not needed.
And this is the super convoluted boilerplate if you want to work with jupytext and run your .py file from parent folder:
import os
import sys
if 'NOTEBOOK_INITIATED_FLAG' not in globals():
NOTEBOOK_INITIATED_FLAG = True
try:
# not in notebook
module_path = os.path.join(os.path.dirname(__file__), os.pardir)
except:
# in notebook
module_path = os.path.abspath(os.path.join('..'))
%cd ..
%pwd
if module_path not in sys.path:
sys.path.append(module_path)
I got annoyed so I wrote a package.
pip install package_root and then you can import package_root # into your PYTHONPATH.
This works by looking through the call stack to determine which file imported package_root and then walking the up directories to find the first directory that is missing an __init__.py file.
So in practice, you’ll need to add __init__.py files to your notebook directories just like you would to get Python packages to work.
If you are working via the Jupyter extension in VS Code, I found that you can now set this path in your settings.json file.
{
"jupyter.notebookFileRoot": "/path/to/your/module/root"
}
Edit: Or, to set it at your workspace root more generically:
{
"jupyter.notebookFileRoot": "${workspaceFolder}"
}