python can't access nonlocal variable before local variable is defined with same name
Question:
I’ve used decorators before and so I was surprised to find a bug in my code:
def make_handler(name, panels):
def get(self):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels = zip(ndb.get_multi(keys), panels)
panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
templates = {'panels': panels, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': get})
The resulting error is:
Traceback (most recent call last):
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:UsersRobertPycharmProjectsbalmoral_doctorsmain.py", line 35, in get
keys = [ndb.Key('Panel', panel) for panel in panels]
UnboundLocalError: local variable 'panels' referenced before assignment
My fix is to change panel to panel2 beyond the first usage:
def make_handler(name, panels):
def get(self):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels2 = zip(ndb.get_multi(keys), panels)
panels2 = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels2]
templates = {'panels': panels2, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': get})
This is my understanding: panels = zip(...) means that panels is a local variable, so the function doesn’t look in the outer scope for panels.
This is done before the get() function is run, and not midway through?
I thought it would firstly grab panels from the outer function, and then when panels gets defined in the inner function after that, it would from then on use the new local panels variable.
Am I on the right track?
Answers:
If a variable is assigned within a function it is assumed to be referring to the local name, unless you explicitly declare it global or, in Python 3.x, nonlocal. If declared global, the variable must be defined in the module’s globals, not covering the above case. You’ve found one Python 2.x solution; another may be to add panels as an argument to get and use functools.partial:
def make_handler(name, panels):
def get(self, panels):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels = zip(ndb.get_multi(keys), panels)
panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
templates = {'panels': panels, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': functools.partial(get, panels=panels)})
See also: Closures in Python, Python nonlocal statement
You are more-or-less correct, and you found the correct resolution. Your problem is equivalent to this:
bars = range(10)
def foo():
thing = [x for x in bars]
bars = 'hello'
foo()
# UnboundLocalError: local variable 'bars' referenced before assignment
At function definition time, it is determined that bars is local scope. And then at function run time, you get the problem that bars hasn’t been assigned.
Yes.
Python’s scoping rules indicate that a function defines a new scope level, and a name is bound to a value in only one scope level in a scope level—it is statically scoped (i.e. all scoping is determined at compilation time). As you understood, you’re trying to violate that by reading from a non-local declaration and writing to a local variable. As you observe, the interpreter objects violently to this by raising an UnboundLocalError: it has understood that panels is a local variable (because it can’t be that and non-local at the same time), but you haven’t assigned (bound) a value to the name, and so it fails.
More technical details
The decision was made in Python to keep track of where variables are at compilation time in the bytecode (to be specific for this case, it’s in a tuple get.__code__.co_varnames for local variables), meaning that a variable can be only used in a single scope level in a certain scope. In Python 2.x, it is not possible to modify a non-local variable; you have either read-only access to a global or non-local variable, or read-write access to a global variable by using the global statement, or read-write access to a local variable (the default). That’s just the way it’s been designed (probably for performance and purity). In Python 3, the nonlocal statement has been introduced with a similar effect to global, but for an intermediate scope.
Binding the modified variable to a different name is the correct solution in this case.
Many people don’t realise this, but Python is actually statically scoped. When Python sees a read from a bare name (i.e. not an attribute of some object), it can determine precisely where that name read will go to purely from compile-time analysis.
If a name is ever assigned to in a function, then that name is a local variable in that function1, and it’s scope extends over the entirety of the function’s body, even over the lines before the assignment.
If the name is not assigned to in a function, then the name is a non-local variable. Python can check the static scope of any surrounding def blocks to see if it’s a local variable in any of those. If not, then that name must be a reference to a module global or a built-in (the choice between global and built-in is resolved dynamically, so you can shadow a built-in with a global that is declared dynamically).
I believe this is done mainly for efficiently. It means that the set of local variables of a function can be known when the bytecode for the function is compiled, and local variable access can be converted into simple index operations. Otherwise Python would have to perform dictionary lookups to access local variables, which would be slower.
So because your get function contains a line of the form panels = ..., then panels is a local variable in the entirety of the body of get. The assignment to keys is looping over the local variable panels before it has been assigned.
1 Unless that name is declared global or nonlocal, but that’s still statically known.
Python compilers first scan the whole code inside a function to determin the set of local variables (passed as prameters or assigned a new val inside the function). That is why you can’t use the variable assigned a new value inside the function before the assignment.
If the compilers didn’t do that then it would have been imposible to create a closure for the function. Closures need the have a reference to all free or non local variables. If a var is free for part of the function and local for the other part….it wouldn’t make sense 🙂
I’ve used decorators before and so I was surprised to find a bug in my code:
def make_handler(name, panels):
def get(self):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels = zip(ndb.get_multi(keys), panels)
panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
templates = {'panels': panels, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': get})
The resulting error is:
Traceback (most recent call last):
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:Program FilesGooglegoogle_appenginelibwebapp2webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:UsersRobertPycharmProjectsbalmoral_doctorsmain.py", line 35, in get
keys = [ndb.Key('Panel', panel) for panel in panels]
UnboundLocalError: local variable 'panels' referenced before assignment
My fix is to change panel to panel2 beyond the first usage:
def make_handler(name, panels):
def get(self):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels2 = zip(ndb.get_multi(keys), panels)
panels2 = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels2]
templates = {'panels': panels2, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': get})
This is my understanding: panels = zip(...) means that panels is a local variable, so the function doesn’t look in the outer scope for panels.
This is done before the get() function is run, and not midway through?
I thought it would firstly grab panels from the outer function, and then when panels gets defined in the inner function after that, it would from then on use the new local panels variable.
Am I on the right track?
If a variable is assigned within a function it is assumed to be referring to the local name, unless you explicitly declare it global or, in Python 3.x, nonlocal. If declared global, the variable must be defined in the module’s globals, not covering the above case. You’ve found one Python 2.x solution; another may be to add panels as an argument to get and use functools.partial:
def make_handler(name, panels):
def get(self, panels):
admin = True
keys = [ndb.Key('Panel', panel) for panel in panels]
panels = zip(ndb.get_multi(keys), panels)
panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
templates = {'panels': panels, 'admin': admin}
self.render_template('panel_page.html', **templates)
return type(name, (BaseHandler,), {'get': functools.partial(get, panels=panels)})
See also: Closures in Python, Python nonlocal statement
You are more-or-less correct, and you found the correct resolution. Your problem is equivalent to this:
bars = range(10)
def foo():
thing = [x for x in bars]
bars = 'hello'
foo()
# UnboundLocalError: local variable 'bars' referenced before assignment
At function definition time, it is determined that bars is local scope. And then at function run time, you get the problem that bars hasn’t been assigned.
Yes.
Python’s scoping rules indicate that a function defines a new scope level, and a name is bound to a value in only one scope level in a scope level—it is statically scoped (i.e. all scoping is determined at compilation time). As you understood, you’re trying to violate that by reading from a non-local declaration and writing to a local variable. As you observe, the interpreter objects violently to this by raising an UnboundLocalError: it has understood that panels is a local variable (because it can’t be that and non-local at the same time), but you haven’t assigned (bound) a value to the name, and so it fails.
More technical details
The decision was made in Python to keep track of where variables are at compilation time in the bytecode (to be specific for this case, it’s in a tuple get.__code__.co_varnames for local variables), meaning that a variable can be only used in a single scope level in a certain scope. In Python 2.x, it is not possible to modify a non-local variable; you have either read-only access to a global or non-local variable, or read-write access to a global variable by using the global statement, or read-write access to a local variable (the default). That’s just the way it’s been designed (probably for performance and purity). In Python 3, the nonlocal statement has been introduced with a similar effect to global, but for an intermediate scope.
Binding the modified variable to a different name is the correct solution in this case.
Many people don’t realise this, but Python is actually statically scoped. When Python sees a read from a bare name (i.e. not an attribute of some object), it can determine precisely where that name read will go to purely from compile-time analysis.
If a name is ever assigned to in a function, then that name is a local variable in that function1, and it’s scope extends over the entirety of the function’s body, even over the lines before the assignment.
If the name is not assigned to in a function, then the name is a non-local variable. Python can check the static scope of any surrounding def blocks to see if it’s a local variable in any of those. If not, then that name must be a reference to a module global or a built-in (the choice between global and built-in is resolved dynamically, so you can shadow a built-in with a global that is declared dynamically).
I believe this is done mainly for efficiently. It means that the set of local variables of a function can be known when the bytecode for the function is compiled, and local variable access can be converted into simple index operations. Otherwise Python would have to perform dictionary lookups to access local variables, which would be slower.
So because your get function contains a line of the form panels = ..., then panels is a local variable in the entirety of the body of get. The assignment to keys is looping over the local variable panels before it has been assigned.
1 Unless that name is declared global or nonlocal, but that’s still statically known.
Python compilers first scan the whole code inside a function to determin the set of local variables (passed as prameters or assigned a new val inside the function). That is why you can’t use the variable assigned a new value inside the function before the assignment.
If the compilers didn’t do that then it would have been imposible to create a closure for the function. Closures need the have a reference to all free or non local variables. If a var is free for part of the function and local for the other part….it wouldn’t make sense 🙂