Imported function not finding filepath when used in jupyter notebooks

Question:

I have the following folder setup:

project
├── pipeline
│   ├── functions.py
|   ├── __init__.py
|   └── library
│       └── data.csv
|
└── notebook.ipynb

I import my functions.py into the notebook as follows:

from pipeline.functions import *

In my notebook I use one of the functions that looks like the following:

def myfunction(df):
    
    libpath = f'library/data.csv'
    try:
        lib_df = pd.read_csv(libpath)
    except OSError:
        print(f'library not found for given filepath: {libpath}')
        sys.exit()
        
    df = df['molecule'].apply(is_covalent, lib_df)
  
    return df

I keep getting a FileNotFoundError for this.

If I try defining the libpath in my notebook and reading it, it works fine.

I have checked i’m in the right directory with os.getcwd() and the pathing is correct.
I’ve also tried different path formats:

libpath = f'pipeline/library/data.csv'
libpath = f'{os.getcwd()}/pipeline/library/data.csv'
libpath = f'pipeline\library\data.csv'
libpath = f'{os.getcwd()}\pipeline\library\data.csv'

None of these appear to work.

I have also tried to explicitly import the function I’m using and this doesn’t make a difference.

Please help.

Asked By: Stuchfield

||

Answers:

Assuming that the function myfunction is in the pipeline/functions script you could use os combined with the __file__ variable to get the correct path as follows.

script_dir = os.path.dirname(os.path.abspath(__file__))
libpath = os.path.join(script_dir, 'library', 'data.csv')

If the function is in the jupyter notebook your libpath should be like this:

libpath = os.path.join(script_dir, 'pipeline', 'library', 'data.csv')

Using the __file__ attribute ensures that the path is always relative to the script and not the current directory where you are axecuting the script from.
Using os.path.join makes sure you have the correct directory separator for your OS (/ or ).

Answered By: Ftagliacarne
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.