Relative import statements, a billion one
Question:
Is there a clear, simple strategy to do relative imports in Python that doesn’t require creating Packages, as I would like to use them between sub-folders of my project while I develop the project?
I’m referring to relative imports, like:
from ..mypackage import mymodule
from ..mypackage.mymodule import mymethod
from .. import mypackage
Only the most simple cases of relative imports actually work. Anything else results in the error message:
ImportError: attempted relative import with no known parent package
Creating a package makes little sense in this context, as I am simply in need of importing relatively between sub-folders in my main project. I use sys.path.append() as a workaround, but that should not be necessary, and it is not portable.
There is a SO post, Relative imports for the billionth time, which addresses this problem. The main article plus 15 answers and 36 comments indicate that this issue has been around for a long time. Many very smart people have offered exotic explanations and proposed cumbersome solutions to such a simple issue.
It seems the issue revolves around where Python thinks the importing module is in the file system at the time it attempts to execute the import statements. My opinion is that Python should be able to figure out if it is and exactly where in the file hierachy that is. I’m looking for a simple way to make import statements work as expected whether the importing module is called from another module, or if it is run from an interpreter, or an IDE like Pycharm.
Answers:
Some SO users may think that this question is "opinion-based" and try to close it. I think this is a valid question, as proven by the fact that the other similar post you linked, Relative imports for the billionth time, addresses the same technical problem and has no exact answer.
I will therefore give you my technical answer, by no means intended to be "the most appropriate solution".
I think that relative imports in Python are badly designed and best avoided1.
My personal solutions are either one of the following.
1. Use importmonkey
If you are ok with an extra package, use importmonkey to allow a sort of "relative imports" without all the faff. This is in a nutshell just a wrapper of sys.path.append, but it makes it more portable.
For example:
root
├─ subfolder
│ └─ subscript.py
└─ rootscript.py
If I have this in rootscript.py:
def somerootfunction():
print("I'm a function at the root of the project")
I can do this from subscript.py to call it:
from importmonkey import add_path; add_path("../")
from rootscript import somerootfunction
somerootfunction()
Not ideal, but it works. Linting and debugging are retained. I find it much simpler, effective, and easy to communicate than "the appropriate solutions"2.
2. Use symbolic links
Alternatively, in limited cases and after having considered a folder restructure or the other alternatives2, use symbolic links.
You can link an entire parent folder placing a link in some sub folder, so Python can finally access it. If you use a relative path during the link creation, this will also make the project portable. Editing the linked code files by opening the items from the link will simply modify the source files. Finally, Git treats symbolic links as just another file that points to a location, so there is no issue with versioning.
The main downside is that this solution is OS-dependent (i.e. a symbolic link created on Linux won’t be read as one if you open your project in Windows). There aren’t workarounds to this, as far as I’m aware. It hasn’t been an issue for my projects.
-
On Linux, I personally use vscode-symlink to make it easy from VSCode. Otherwise just use ln -s source_path destination_path.
-
On Windows, you can use mklink. If you’re using it from a non-cmd terminal (e.g. the VSCode terminal) remember to prepend cmd /c before using it. E.g., if I’m in the sub-folder and I want to link to a parent folder file, cmd /c mklink rootscript.py ..rootscript.py.
You can link to a directory with the /D argument.
1 I think this is proven by the fact that there are many proposed solutions and much confusion on the subject. I’m listing some in note 2 below. As a personal addendum, I do believe that this is another proof that Python is a good and easy scripting language, but a flawed and hard to learn programming language. I don’t think it should be used for building large programs.
For my large software projects I use other languages, and if I need Python functions or utilities (likely e.g. when dealing with Machine Learning), I use interop methods to call separate, small Python scripts.
2 Probably, "the appropriate" solution is to be considered installing the project with a pyproject.toml file and/or using flit/poetry. This is what experienced people would probably suggest. However, there is still a lot of reading and learning required to do it. Also, there is much confusion on this solution, due to this having replaced older standard solutions. Others believe that pip install with editable is sufficient. Some suggest editing PYTHONPATH, which I think is downright bad.
I think that these solutions are viable but complicate life immensely, especially for newcomers. Some of these will actually work but introduce issues or complications with either debugging and/or linting the import source, making the programming experience far from ideal.
Is there a clear, simple strategy to do relative imports in Python that doesn’t require creating Packages, as I would like to use them between sub-folders of my project while I develop the project?
I’m referring to relative imports, like:
from ..mypackage import mymodule
from ..mypackage.mymodule import mymethod
from .. import mypackage
Only the most simple cases of relative imports actually work. Anything else results in the error message:
ImportError: attempted relative import with no known parent package
Creating a package makes little sense in this context, as I am simply in need of importing relatively between sub-folders in my main project. I use sys.path.append() as a workaround, but that should not be necessary, and it is not portable.
There is a SO post, Relative imports for the billionth time, which addresses this problem. The main article plus 15 answers and 36 comments indicate that this issue has been around for a long time. Many very smart people have offered exotic explanations and proposed cumbersome solutions to such a simple issue.
It seems the issue revolves around where Python thinks the importing module is in the file system at the time it attempts to execute the import statements. My opinion is that Python should be able to figure out if it is and exactly where in the file hierachy that is. I’m looking for a simple way to make import statements work as expected whether the importing module is called from another module, or if it is run from an interpreter, or an IDE like Pycharm.
Some SO users may think that this question is "opinion-based" and try to close it. I think this is a valid question, as proven by the fact that the other similar post you linked, Relative imports for the billionth time, addresses the same technical problem and has no exact answer.
I will therefore give you my technical answer, by no means intended to be "the most appropriate solution".
I think that relative imports in Python are badly designed and best avoided1.
My personal solutions are either one of the following.
1. Use importmonkey
If you are ok with an extra package, use importmonkey to allow a sort of "relative imports" without all the faff. This is in a nutshell just a wrapper of sys.path.append, but it makes it more portable.
For example:
root
├─ subfolder
│ └─ subscript.py
└─ rootscript.py
If I have this in rootscript.py:
def somerootfunction():
print("I'm a function at the root of the project")
I can do this from subscript.py to call it:
from importmonkey import add_path; add_path("../")
from rootscript import somerootfunction
somerootfunction()
Not ideal, but it works. Linting and debugging are retained. I find it much simpler, effective, and easy to communicate than "the appropriate solutions"2.
2. Use symbolic links
Alternatively, in limited cases and after having considered a folder restructure or the other alternatives2, use symbolic links.
You can link an entire parent folder placing a link in some sub folder, so Python can finally access it. If you use a relative path during the link creation, this will also make the project portable. Editing the linked code files by opening the items from the link will simply modify the source files. Finally, Git treats symbolic links as just another file that points to a location, so there is no issue with versioning.
The main downside is that this solution is OS-dependent (i.e. a symbolic link created on Linux won’t be read as one if you open your project in Windows). There aren’t workarounds to this, as far as I’m aware. It hasn’t been an issue for my projects.
-
On Linux, I personally use vscode-symlink to make it easy from VSCode. Otherwise just use
ln -s source_path destination_path. -
On Windows, you can use
mklink. If you’re using it from a non-cmd terminal (e.g. the VSCode terminal) remember to prependcmd /cbefore using it. E.g., if I’m in the sub-folder and I want to link to a parent folder file,cmd /c mklink rootscript.py ..rootscript.py.
You can link to a directory with the/Dargument.
1 I think this is proven by the fact that there are many proposed solutions and much confusion on the subject. I’m listing some in note 2 below. As a personal addendum, I do believe that this is another proof that Python is a good and easy scripting language, but a flawed and hard to learn programming language. I don’t think it should be used for building large programs.
For my large software projects I use other languages, and if I need Python functions or utilities (likely e.g. when dealing with Machine Learning), I use interop methods to call separate, small Python scripts.
2 Probably, "the appropriate" solution is to be considered installing the project with a pyproject.toml file and/or using flit/poetry. This is what experienced people would probably suggest. However, there is still a lot of reading and learning required to do it. Also, there is much confusion on this solution, due to this having replaced older standard solutions. Others believe that pip install with editable is sufficient. Some suggest editing PYTHONPATH, which I think is downright bad.
I think that these solutions are viable but complicate life immensely, especially for newcomers. Some of these will actually work but introduce issues or complications with either debugging and/or linting the import source, making the programming experience far from ideal.