Python class scoping rules
Question:
EDIT: Looks like this is a very old “bug” or, actually, feature. See, e.g., this mail
I am trying to understand the Python scoping rules. More precisely, I thought that I understand them but then I found this code here:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
class C:
print(x)
print(y)
y = 1
func()
In Python 3.4 the output is:
xlocal
ytop
If I replace the inner class by a function then it reasonably gives UnboundLocalError. Could you explain me why it behaves this strange way with classes and what is the reason for such choice of scoping rules?
Answers:
First focus on the case of a closure — a function within a function:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
def inner():
# global y
print(x)
print(y)
y='inner y'
print(y)
inner()
Note the commented out global in inner If you run this, it replicates the UnboundLocalError you got. Why?
Run dis.dis on it:
>>> import dis
>>> dis.dis(func)
6 0 LOAD_CONST 1 ('xlocal')
3 STORE_DEREF 0 (x)
7 6 LOAD_CONST 2 ('ylocal')
9 STORE_FAST 0 (y)
8 12 LOAD_CLOSURE 0 (x)
15 BUILD_TUPLE 1
18 LOAD_CONST 3 (<code object inner at 0x101500270, file "Untitled 3.py", line 8>)
21 LOAD_CONST 4 ('func.<locals>.inner')
24 MAKE_CLOSURE 0
27 STORE_FAST 1 (inner)
14 30 LOAD_FAST 1 (inner)
33 CALL_FUNCTION 0 (0 positional, 0 keyword pair)
36 POP_TOP
37 LOAD_CONST 0 (None)
40 RETURN_VALUE
Note the different access mode of x vs y inside of func. The use of y='inner y' inside of inner has created the UnboundLocalError
Now uncomment global y inside of inner. Now you have unambiguously create y to be the top global version until resigned as y='inner y'
With global uncommented, prints:
xlocal
ytop
inner y
You can get a more sensible result with:
x = "xtop"
y = "ytop"
def func():
global y, x
print(x,y)
x = "xlocal"
y = "ylocal"
def inner():
global y
print(x,y)
y = 'inner y'
print(x,y)
inner()
Prints:
xtop ytop
xlocal ylocal
xlocal inner y
The analysis of the closure class is complicated by instance vs class variables and what / when a naked class (with no instance) is being executed.
The bottom line is the same: If you reference a name outside the local namespace and then assign to the same name locally you get a surprising result.
The ‘fix’ is the same: use the global keyword:
x = "xtop"
y = "ytop"
def func():
global x, y
x = "xlocal"
y = "ylocal"
class Inner:
print(x, y)
y = 'Inner_y'
print(x, y)
Prints:
xlocal ylocal
xlocal Inner_y
You can read more about Python 3 scope rules in PEP 3104
TL;DR: This behaviour has existed since Python 2.1 PEP 227: Nested Scopes, and was known back then. If a name is assigned to within a class body (like y), then it is assumed to be a local/global variable; if it is not assigned to (x), then it also can potentially point to a closure cell. The lexical variables do not show up as local/global names to the class body.
On Python 3.4, dis.dis(func) shows the following:
>>> dis.dis(func)
4 0 LOAD_CONST 1 ('xlocal')
3 STORE_DEREF 0 (x)
5 6 LOAD_CONST 2 ('ylocal')
9 STORE_FAST 0 (y)
6 12 LOAD_BUILD_CLASS
13 LOAD_CLOSURE 0 (x)
16 BUILD_TUPLE 1
19 LOAD_CONST 3 (<code object C at 0x7f083c9bbf60, file "test.py", line 6>)
22 LOAD_CONST 4 ('C')
25 MAKE_CLOSURE 0
28 LOAD_CONST 4 ('C')
31 CALL_FUNCTION 2 (2 positional, 0 keyword pair)
34 STORE_FAST 1 (C)
37 LOAD_CONST 0 (None)
40 RETURN_VALUE
The LOAD_BUILD_CLASS loads the builtins.__build_class__ on the stack; this is called with arguments __build_class__(func, name); where func is the class body, and name is 'C'. The class body is the constant #3 for the function func:
>>> dis.dis(func.__code__.co_consts[3])
6 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
6 LOAD_CONST 0 ('func.<locals>.C')
9 STORE_NAME 2 (__qualname__)
7 12 LOAD_NAME 3 (print)
15 LOAD_CLASSDEREF 0 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 POP_TOP
8 22 LOAD_NAME 3 (print)
25 LOAD_NAME 4 (y)
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 POP_TOP
9 32 LOAD_CONST 1 (1)
35 STORE_NAME 4 (y)
38 LOAD_CONST 2 (None)
41 RETURN_VALUE
Within the class body, x is accessed with LOAD_CLASSDEREF (15) while y is load with LOAD_NAME (25). The LOAD_CLASSDEREF is a Python 3.4+ opcode for loading values from closure cells specifically within class bodies (in previous versions, the generic LOAD_DEREF was used); the LOAD_NAME is for loading values from locals and then globals. However closure cells show up neither as locals nor globals.
Now, because the name y is stored to within the class body (35), it is consistently being used as not a closure cell but a local/global name.
The closure cells do not show up as local variables to the class body.
This behaviour has been true ever since implementing PEP 227 – nested scopes. And back then BDFL stated that this should not be fixed – and thus it has been for these 13+ years.
The only change since PEP 227 is the addition of nonlocal in Python 3; if one uses it within the class body, the class body can set the values of the cells within the containing scope:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
class C:
nonlocal y # y here now refers to the outer variable
print(x)
print(y)
y = 1
print(y)
print(C.y)
func()
The output now is
xlocal
ylocal
1
Traceback (most recent call last):
File "test.py", line 15, in <module>
func()
File "test.py", line 13, in func
print(C.y)
AttributeError: type object 'C' has no attribute 'y'
That is, print(y) read the value of the cell y of the containing scope, and y = 1 set the value in that cell; in this case, no attribute was created for the class C.
EDIT: Looks like this is a very old “bug” or, actually, feature. See, e.g., this mail
I am trying to understand the Python scoping rules. More precisely, I thought that I understand them but then I found this code here:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
class C:
print(x)
print(y)
y = 1
func()
In Python 3.4 the output is:
xlocal
ytop
If I replace the inner class by a function then it reasonably gives UnboundLocalError. Could you explain me why it behaves this strange way with classes and what is the reason for such choice of scoping rules?
First focus on the case of a closure — a function within a function:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
def inner():
# global y
print(x)
print(y)
y='inner y'
print(y)
inner()
Note the commented out global in inner If you run this, it replicates the UnboundLocalError you got. Why?
Run dis.dis on it:
>>> import dis
>>> dis.dis(func)
6 0 LOAD_CONST 1 ('xlocal')
3 STORE_DEREF 0 (x)
7 6 LOAD_CONST 2 ('ylocal')
9 STORE_FAST 0 (y)
8 12 LOAD_CLOSURE 0 (x)
15 BUILD_TUPLE 1
18 LOAD_CONST 3 (<code object inner at 0x101500270, file "Untitled 3.py", line 8>)
21 LOAD_CONST 4 ('func.<locals>.inner')
24 MAKE_CLOSURE 0
27 STORE_FAST 1 (inner)
14 30 LOAD_FAST 1 (inner)
33 CALL_FUNCTION 0 (0 positional, 0 keyword pair)
36 POP_TOP
37 LOAD_CONST 0 (None)
40 RETURN_VALUE
Note the different access mode of x vs y inside of func. The use of y='inner y' inside of inner has created the UnboundLocalError
Now uncomment global y inside of inner. Now you have unambiguously create y to be the top global version until resigned as y='inner y'
With global uncommented, prints:
xlocal
ytop
inner y
You can get a more sensible result with:
x = "xtop"
y = "ytop"
def func():
global y, x
print(x,y)
x = "xlocal"
y = "ylocal"
def inner():
global y
print(x,y)
y = 'inner y'
print(x,y)
inner()
Prints:
xtop ytop
xlocal ylocal
xlocal inner y
The analysis of the closure class is complicated by instance vs class variables and what / when a naked class (with no instance) is being executed.
The bottom line is the same: If you reference a name outside the local namespace and then assign to the same name locally you get a surprising result.
The ‘fix’ is the same: use the global keyword:
x = "xtop"
y = "ytop"
def func():
global x, y
x = "xlocal"
y = "ylocal"
class Inner:
print(x, y)
y = 'Inner_y'
print(x, y)
Prints:
xlocal ylocal
xlocal Inner_y
You can read more about Python 3 scope rules in PEP 3104
TL;DR: This behaviour has existed since Python 2.1 PEP 227: Nested Scopes, and was known back then. If a name is assigned to within a class body (like y), then it is assumed to be a local/global variable; if it is not assigned to (x), then it also can potentially point to a closure cell. The lexical variables do not show up as local/global names to the class body.
On Python 3.4, dis.dis(func) shows the following:
>>> dis.dis(func)
4 0 LOAD_CONST 1 ('xlocal')
3 STORE_DEREF 0 (x)
5 6 LOAD_CONST 2 ('ylocal')
9 STORE_FAST 0 (y)
6 12 LOAD_BUILD_CLASS
13 LOAD_CLOSURE 0 (x)
16 BUILD_TUPLE 1
19 LOAD_CONST 3 (<code object C at 0x7f083c9bbf60, file "test.py", line 6>)
22 LOAD_CONST 4 ('C')
25 MAKE_CLOSURE 0
28 LOAD_CONST 4 ('C')
31 CALL_FUNCTION 2 (2 positional, 0 keyword pair)
34 STORE_FAST 1 (C)
37 LOAD_CONST 0 (None)
40 RETURN_VALUE
The LOAD_BUILD_CLASS loads the builtins.__build_class__ on the stack; this is called with arguments __build_class__(func, name); where func is the class body, and name is 'C'. The class body is the constant #3 for the function func:
>>> dis.dis(func.__code__.co_consts[3])
6 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
6 LOAD_CONST 0 ('func.<locals>.C')
9 STORE_NAME 2 (__qualname__)
7 12 LOAD_NAME 3 (print)
15 LOAD_CLASSDEREF 0 (x)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 POP_TOP
8 22 LOAD_NAME 3 (print)
25 LOAD_NAME 4 (y)
28 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
31 POP_TOP
9 32 LOAD_CONST 1 (1)
35 STORE_NAME 4 (y)
38 LOAD_CONST 2 (None)
41 RETURN_VALUE
Within the class body, x is accessed with LOAD_CLASSDEREF (15) while y is load with LOAD_NAME (25). The LOAD_CLASSDEREF is a Python 3.4+ opcode for loading values from closure cells specifically within class bodies (in previous versions, the generic LOAD_DEREF was used); the LOAD_NAME is for loading values from locals and then globals. However closure cells show up neither as locals nor globals.
Now, because the name y is stored to within the class body (35), it is consistently being used as not a closure cell but a local/global name.
The closure cells do not show up as local variables to the class body.
This behaviour has been true ever since implementing PEP 227 – nested scopes. And back then BDFL stated that this should not be fixed – and thus it has been for these 13+ years.
The only change since PEP 227 is the addition of nonlocal in Python 3; if one uses it within the class body, the class body can set the values of the cells within the containing scope:
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
class C:
nonlocal y # y here now refers to the outer variable
print(x)
print(y)
y = 1
print(y)
print(C.y)
func()
The output now is
xlocal
ylocal
1
Traceback (most recent call last):
File "test.py", line 15, in <module>
func()
File "test.py", line 13, in func
print(C.y)
AttributeError: type object 'C' has no attribute 'y'
That is, print(y) read the value of the cell y of the containing scope, and y = 1 set the value in that cell; in this case, no attribute was created for the class C.