Effect of using sys.path.insert(0, path) and sys.path(append) when loading modules

Question

I was recently having a problem with a python ImportError, where the module was found when running on my local computer but not found on the CI server. I solved this problem by swapping sys.path.append(path) in my script with sys.path.insert(0, path) where path is the string module location.

Since this is my module and not an installed package (related question), why does the order of paths fix this problem?

Asked By: Michael Barton

||

Source

Answer 1

Because python checks in the directories in sequential order starting at the first directory in sys.path list, till it find the .py file it was looking for.

Ideally, the current directory or the directory of the script is the first always the first element in the list, unless you modify it, like you did. From documentation –

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.

So, most probably, you had a .py file with the same name as the module you were trying to import from, in the current directory (where the script was being run from).

Also, a thing to note about ImportErrors , lets say the import error says –
ImportError: No module named main – it doesn’t mean the main.py is overwritten, no if that was overwritten we would not be having issues trying to read it. Its some module above this that got overwritten with a .py or some other file.

Example –

My directory structure looks like –

 - test
    - shared
         - __init__.py
         - phtest.py
  - testmain.py

Now From testmain.py , I call from shared import phtest , it works fine.

Now lets say I introduce a shared.py in test directory` , example –

 - test
    - shared
         - __init__.py
         - phtest.py
  - testmain.py 
  - shared.py

Now when I try to do from shared import phtest from testmain.py , I will get the error –

ImportError: cannot import name 'phtest'

As you can see above, the file that is causing the issue is shared.py , not phtest.py .

Answered By: Anand S Kumar

Answer 2

I’m quite a beginner in Python and I found the answer of Anand was very good but quite complicated to me, so I try to reformulate :

1) insert and append methods are not specific to sys.path and as in other languages they add an item into a list or array and :
* append(item) add item to the end of the list,
* insert(n, item) inserts the item at the nth position in the list (0 at the beginning, 1 after the first element, etc …).

2) As Anand said, python search the import files in each directory of the path in the order of the path, so :
* If you have no file name collisions, the order of the path has no impact,
* If you look after a function already defined in the path and you use append to add your path, you will not get your function but the predefined one.

But I think that it is better to use append and not insert to not overload the standard behaviour of Python, and use non-ambiguous names for your files and methods.

Answered By: herve-guerin

Effect of using sys.path.insert(0, path) and sys.path(append) when loading modules

Question:

Answers: