Cyclic module dependencies and relative imports in Python
Question:
Suppose we have two modules with cyclic dependencies:
# a.py
import b
def f(): return b.y
x = 42
# b.py
import a
def g(): return a.x
y = 43
The two modules are in the directory pkg with an empty __init__.py. Importing pkg.a or pkg.b works fine, as explained in this answer. If I change the imports to relative imports
from . import b
I get an ImportError when trying to import one of the modules:
>>> import pkg.a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pkg/a.py", line 1, in <module>
from . import b
File "pkg/b.py", line 1, in <module>
from . import a
ImportError: cannot import name a
Why do I get this error? Isn’t the situation pretty much the same as above? (Is this related to this question?)
Edit: This question is not about software design. I’m aware of ways to avoid the circular dependency, but I’m interested in the reason for the error anyway.
Answers:
(Incedentally, the relative import doesn’t matter. Using from pkg import… reveals the same exception.)
I think what’s going on here is that the difference between from foo import bar and import foo.bar is that in the first one, the value bar could be module in pkg foo or it could be a variable in a module foo. In the second case, it’s invalid for bar to be anything but a module/package.
This would matter because if bar is known to be a module, then the contents of sys.modules is sufficient to populate it. If it might be a variable in the foo module, then the interpreter must actually look into the contents of foo, but while importing foo, that would be invalid; the actual module has not been populated yet.
In the case of a relative import, we understand from . import bar to mean import the bar module from the package that contains the current module, but this is really just syntactic sugar, the . name is translated to a fully qualified name and passed to __import__(), and thus it looks for all the world like the ambigious from foo import bar
First let’s start with how from import work in python:
Well first let’s look at the byte code:
>>> def foo():
... from foo import bar
>>> dis.dis(foo)
2 0 LOAD_CONST 1 (-1)
3 LOAD_CONST 2 (('bar',))
6 IMPORT_NAME 0 (foo)
9 IMPORT_FROM 1 (bar)
12 STORE_FAST 0 (bar)
15 POP_TOP
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
hmm interesting :), so from foo import bar is translated to first IMPORT_NAME foo which equivalent to import foo and then IMPORT_FROM bar.
Now what IMPORT_FROM do ?
let’s see what python do when he found IMPORT_FROM:
TARGET(IMPORT_FROM)
w = GETITEM(names, oparg);
v = TOP();
READ_TIMESTAMP(intr0);
x = import_from(v, w);
READ_TIMESTAMP(intr1);
PUSH(x);
if (x != NULL) DISPATCH();
break;
Well basically he get the the names to import from, which is in our foo() function will be bar, then he pop from the frame stack the value v which is the return of the the last opcode executed which is IMPORT_NAME, then call the function import_from() with this two arguments :
static PyObject *
import_from(PyObject *v, PyObject *name)
{
PyObject *x;
x = PyObject_GetAttr(v, name);
if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
PyErr_Format(PyExc_ImportError, "cannot import name %S", name);
}
return x;
}
As you can see the import_from() function is quiet easy, it try first to get the attribute name from the module v, if it don’t exist it raise ImportError else return this attribute.
Now what this have to do with relative import ?
Well relative import like from . import b are equivalent for example in the case that is in the OP question to from pkg import b.
But how this happen ? To understand this we should take a look to the import.c module of python specially to the function get_parent(). As you see the function is quiet long to list here but in general what it does when it see a relative import is to try to replace the dot . with the parent package depending on the __main__ module, which is again from the OP question is the package pkg.
Now let’s put all this together and try to figure out why we end up with the behavior in the OP question.
For this it will help us if we can see what python do when doing imports, well it’s our lucky day python come already with this feature which can be enabled by running it in extra verbose mode -vv.
So using the command line: python -vv -c 'import pkg.b':
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import pkg # directory pkg
# trying pkg/__init__.so
# trying pkg/__init__module.so
# trying pkg/__init__.py
# pkg/__init__.pyc matches pkg/__init__.py
import pkg # precompiled from pkg/__init__.pyc
# trying pkg/b.so
# trying pkg/bmodule.so
# trying pkg/b.py
# pkg/b.pyc matches pkg/b.py
import pkg.b # precompiled from pkg/b.pyc
# trying pkg/a.so
# trying pkg/amodule.so
# trying pkg/a.py
# pkg/a.pyc matches pkg/a.py
import pkg.a # precompiled from pkg/a.pyc
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "pkg/b.py", line 1, in <module>
from . import a
File "pkg/a.py", line 2, in <module>
from . import a
ImportError: cannot import name a
# clear __builtin__._
hmm what just happen before the ImportError ?
First) from . import a in pkg/b.py is called, which is translated as explained above to from pkg import a, which is again in bytecode is equivalent to import pkg; getattr(pkg, 'a'). But wait a minute a is a module too ?!
Well here came the fun part if we have something like from module|package import module in this case a second import will happen which is the import of the module in the import clause. So again in the OP example we need now to import pkg/a.py, and as you know first of all we set in our sys.modules a key for our new module which will be pkg.a and then we continue our interpretation of the module pkg/a.py, but before the module pkg/a.py finish importing it call from . import b.
Now come the Second) part, pkg/b.py will be imported and in it turn it will first attempt to import pkg which because pkg is already imported so there is a key pkg in our sys.modules it will just return the value of that key. Then it will import b set the pkg.b key in sys.modules and start the interpretation. And we arrive to this line from . import a !
But remember pkg/a.py is already imported which mean ('pkg.a' in sys.modules) == True so the import will be skipped, and only the getattr(pkg, 'a') will be called , but what will happen ? python didn’t finish importing pkg/a.py !? So only getattr(pkg, 'a') will be called , and this will raise an AttributeError in the import_from() function, which will be translated to ImportError(cannot import name a).
DISCLAIM : This is my own effort to understand what is happening inside the interpreter, i’m far away of being an expert.
EDIt: This answer was rephrased because when i tried to read it again i remarked how my answer was bad formulated, hope now it will be more useful 🙂
As an additional Note:
I had the following module Structure:
base
+guiStuff
-gui
+databaseStuff
-db
-basescript
I wanted to be able to run my script by import base.basescript, however this failed with an error since the gui file had an import base.databaseStuff.db which caused an import of base. Since base was only registered as __main__ it caused a second execution of the whole imports and the error above unless I used an external script above base thus importing base / basescript only once. To prevent this I put the following in my base script:
if __name__ == '__main__' or
not '__main__' in sys.modules or
sys.modules['__main__'].__file__ != __file__:
#imports here
Suppose we have two modules with cyclic dependencies:
# a.py
import b
def f(): return b.y
x = 42
# b.py
import a
def g(): return a.x
y = 43
The two modules are in the directory pkg with an empty __init__.py. Importing pkg.a or pkg.b works fine, as explained in this answer. If I change the imports to relative imports
from . import b
I get an ImportError when trying to import one of the modules:
>>> import pkg.a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pkg/a.py", line 1, in <module>
from . import b
File "pkg/b.py", line 1, in <module>
from . import a
ImportError: cannot import name a
Why do I get this error? Isn’t the situation pretty much the same as above? (Is this related to this question?)
Edit: This question is not about software design. I’m aware of ways to avoid the circular dependency, but I’m interested in the reason for the error anyway.
(Incedentally, the relative import doesn’t matter. Using from pkg import… reveals the same exception.)
I think what’s going on here is that the difference between from foo import bar and import foo.bar is that in the first one, the value bar could be module in pkg foo or it could be a variable in a module foo. In the second case, it’s invalid for bar to be anything but a module/package.
This would matter because if bar is known to be a module, then the contents of sys.modules is sufficient to populate it. If it might be a variable in the foo module, then the interpreter must actually look into the contents of foo, but while importing foo, that would be invalid; the actual module has not been populated yet.
In the case of a relative import, we understand from . import bar to mean import the bar module from the package that contains the current module, but this is really just syntactic sugar, the . name is translated to a fully qualified name and passed to __import__(), and thus it looks for all the world like the ambigious from foo import bar
First let’s start with how from import work in python:
Well first let’s look at the byte code:
>>> def foo():
... from foo import bar
>>> dis.dis(foo)
2 0 LOAD_CONST 1 (-1)
3 LOAD_CONST 2 (('bar',))
6 IMPORT_NAME 0 (foo)
9 IMPORT_FROM 1 (bar)
12 STORE_FAST 0 (bar)
15 POP_TOP
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
hmm interesting :), so from foo import bar is translated to first IMPORT_NAME foo which equivalent to import foo and then IMPORT_FROM bar.
Now what IMPORT_FROM do ?
let’s see what python do when he found IMPORT_FROM:
TARGET(IMPORT_FROM)
w = GETITEM(names, oparg);
v = TOP();
READ_TIMESTAMP(intr0);
x = import_from(v, w);
READ_TIMESTAMP(intr1);
PUSH(x);
if (x != NULL) DISPATCH();
break;
Well basically he get the the names to import from, which is in our foo() function will be bar, then he pop from the frame stack the value v which is the return of the the last opcode executed which is IMPORT_NAME, then call the function import_from() with this two arguments :
static PyObject *
import_from(PyObject *v, PyObject *name)
{
PyObject *x;
x = PyObject_GetAttr(v, name);
if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
PyErr_Format(PyExc_ImportError, "cannot import name %S", name);
}
return x;
}
As you can see the import_from() function is quiet easy, it try first to get the attribute name from the module v, if it don’t exist it raise ImportError else return this attribute.
Now what this have to do with relative import ?
Well relative import like from . import b are equivalent for example in the case that is in the OP question to from pkg import b.
But how this happen ? To understand this we should take a look to the import.c module of python specially to the function get_parent(). As you see the function is quiet long to list here but in general what it does when it see a relative import is to try to replace the dot . with the parent package depending on the __main__ module, which is again from the OP question is the package pkg.
Now let’s put all this together and try to figure out why we end up with the behavior in the OP question.
For this it will help us if we can see what python do when doing imports, well it’s our lucky day python come already with this feature which can be enabled by running it in extra verbose mode -vv.
So using the command line: python -vv -c 'import pkg.b':
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import pkg # directory pkg
# trying pkg/__init__.so
# trying pkg/__init__module.so
# trying pkg/__init__.py
# pkg/__init__.pyc matches pkg/__init__.py
import pkg # precompiled from pkg/__init__.pyc
# trying pkg/b.so
# trying pkg/bmodule.so
# trying pkg/b.py
# pkg/b.pyc matches pkg/b.py
import pkg.b # precompiled from pkg/b.pyc
# trying pkg/a.so
# trying pkg/amodule.so
# trying pkg/a.py
# pkg/a.pyc matches pkg/a.py
import pkg.a # precompiled from pkg/a.pyc
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "pkg/b.py", line 1, in <module>
from . import a
File "pkg/a.py", line 2, in <module>
from . import a
ImportError: cannot import name a
# clear __builtin__._
hmm what just happen before the ImportError ?
First) from . import a in pkg/b.py is called, which is translated as explained above to from pkg import a, which is again in bytecode is equivalent to import pkg; getattr(pkg, 'a'). But wait a minute a is a module too ?!
Well here came the fun part if we have something like from module|package import module in this case a second import will happen which is the import of the module in the import clause. So again in the OP example we need now to import pkg/a.py, and as you know first of all we set in our sys.modules a key for our new module which will be pkg.a and then we continue our interpretation of the module pkg/a.py, but before the module pkg/a.py finish importing it call from . import b.
Now come the Second) part, pkg/b.py will be imported and in it turn it will first attempt to import pkg which because pkg is already imported so there is a key pkg in our sys.modules it will just return the value of that key. Then it will import b set the pkg.b key in sys.modules and start the interpretation. And we arrive to this line from . import a !
But remember pkg/a.py is already imported which mean ('pkg.a' in sys.modules) == True so the import will be skipped, and only the getattr(pkg, 'a') will be called , but what will happen ? python didn’t finish importing pkg/a.py !? So only getattr(pkg, 'a') will be called , and this will raise an AttributeError in the import_from() function, which will be translated to ImportError(cannot import name a).
DISCLAIM : This is my own effort to understand what is happening inside the interpreter, i’m far away of being an expert.
EDIt: This answer was rephrased because when i tried to read it again i remarked how my answer was bad formulated, hope now it will be more useful 🙂
As an additional Note:
I had the following module Structure:
base
+guiStuff
-gui
+databaseStuff
-db
-basescript
I wanted to be able to run my script by import base.basescript, however this failed with an error since the gui file had an import base.databaseStuff.db which caused an import of base. Since base was only registered as __main__ it caused a second execution of the whole imports and the error above unless I used an external script above base thus importing base / basescript only once. To prevent this I put the following in my base script:
if __name__ == '__main__' or
not '__main__' in sys.modules or
sys.modules['__main__'].__file__ != __file__:
#imports here