Python lambda closure scoping
Question:
I am trying to use closures to eliminate a variable from a function signature (the application is to make writing all the functions needed for connecting Qt signals for an interface to control a largish number of parameters to the dictionary that stores the values ).
I do not understand why the case of using the lambda not wrapped in another function returns the last name for all cases.
names = ['a', 'b', 'c']
def test_fun(name, x):
print(name, x)
def gen_clousure(name):
return lambda x: test_fun(name, x)
funcs1 = [gen_clousure(n) for n in names]
funcs2 = [lambda x: test_fun(n, x) for n in names]
# this is what I want
In [88]: for f in funcs1:
....: f(1)
a 1
b 1
c 1
# I do not understand why I get this
In [89]: for f in funcs2:
....: f(1)
c 1
c 1
c 1
Answers:
The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.
You say at the beginning that you want to “use closures to eliminate a variable from a function signature”, but it doesn’t really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by “eliminate”.) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a “snapshot” of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:
>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
In the second example, passing n as an argument to the function “locks in” the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn’t work this way, things like global variables wouldn’t work at all; it’s essential that free variables be looked up at the time of use.)
Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.
If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:
>>> def makeFunc(n):
... return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that “saves” the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own “private” variable called x. (For this simple case, this can also be done using a lambda for the outer function — stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] — but this is less readable.)
Notice that you still have to pass your n as an argument, it’s just that, by doing it this way, you don’t pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is “clean” and doesn’t have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.
The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).
I am trying to use closures to eliminate a variable from a function signature (the application is to make writing all the functions needed for connecting Qt signals for an interface to control a largish number of parameters to the dictionary that stores the values ).
I do not understand why the case of using the lambda not wrapped in another function returns the last name for all cases.
names = ['a', 'b', 'c']
def test_fun(name, x):
print(name, x)
def gen_clousure(name):
return lambda x: test_fun(name, x)
funcs1 = [gen_clousure(n) for n in names]
funcs2 = [lambda x: test_fun(n, x) for n in names]
# this is what I want
In [88]: for f in funcs1:
....: f(1)
a 1
b 1
c 1
# I do not understand why I get this
In [89]: for f in funcs2:
....: f(1)
c 1
c 1
c 1
The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.
You say at the beginning that you want to “use closures to eliminate a variable from a function signature”, but it doesn’t really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by “eliminate”.) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a “snapshot” of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:
>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
In the second example, passing n as an argument to the function “locks in” the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn’t work this way, things like global variables wouldn’t work at all; it’s essential that free variables be looked up at the time of use.)
Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.
If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:
>>> def makeFunc(n):
... return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that “saves” the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own “private” variable called x. (For this simple case, this can also be done using a lambda for the outer function — stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] — but this is less readable.)
Notice that you still have to pass your n as an argument, it’s just that, by doing it this way, you don’t pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is “clean” and doesn’t have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.
The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).