Lambdas from a list comprehension are returning a lambda when called
Question:
I am trying to iterate the lambda func over a list as in test.py, and I want to get the call result of the lambda, not the function object itself. However, the following output really confused me.
------test.py---------
#!/bin/env python
#coding: utf-8
a = [lambda: i for i in range(5)]
for i in a:
print i()
--------output---------
<function <lambda> at 0x7f489e542e60>
<function <lambda> at 0x7f489e542ed8>
<function <lambda> at 0x7f489e542f50>
<function <lambda> at 0x7f489e54a050>
<function <lambda> at 0x7f489e54a0c8>
I modified the variable name when print the call result to t as following, and everything goes well. I am wondering what is all about of that. ?
--------test.py(update)--------
a = [lambda: i for i in range(5)]
for t in a:
print t()
-----------output-------------
4
4
4
4
4
This behaviour is 2.x-specific, and is a special case of the problem described at What do lambda function closures capture?. In 3.x, the list comprehension creates its own scope for the iteration variable, so it isn’t modifiable from outside. It is, however, still late binding, so each print will produce 4.
Answers:
Closures in Python are late-binding, meaning that each lambda function in the list will only evaluate the variable i when invoked, and not when defined. That’s why all functions return the same value, i.e. the last value of ì (which is 4).
To avoid this, one technique is to bind the value of i to a local named parameter:
>>> a = [lambda i=i: i for i in range(5)]
>>> for t in a:
... print t()
...
0
1
2
3
4
Another option is to create a partial function and bind the current value of i as its parameter:
>>> from functools import partial
>>> a = [partial(lambda x: x, i) for i in range(5)]
>>> for t in a:
... print t()
...
0
1
2
3
4
Edit: Sorry, misread the question initially, since these kind of questions are so often about late binding (thanks @soon for the comment).
The second reason for the behavior is list comprehension’s variable leaking in Python2 as others have already explained. When using i as the iteration variable in the for loop, each function prints the current value of i (for the reasons stated above), which is simply the function itself.
When using a different name (e.g. t), functions print the last value of i as it was in the list comprehension loop, which is 4.
lambda: i is an anonymous function with no arguments that returns i. So you are generating a list of anonymous functions, which you can later (in the second example) bind to the name t and invoke with (). Note you can do the same with non-anonymous functions:
>>> def f():
... return 42
...
>>> name = f
>>> name
<function f at 0x7fed4d70fb90>
>>> name()
42
@plamut has just answered the implied other part of the question, so I won’t.
In Python 2 list comprehension ‘leaks’ the variables to outer scope:
>>> [i for i in xrange(3)]
[0, 1, 2]
>>> i
2
Note that the behavior is different on Python 3:
>>> [i for i in range(3)]
[0, 1, 2]
>>> i
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'i' is not defined
When you define lambda it’s bound to variable i, not its’ current value as your second example shows.
Now when you assign new value to i the lambda will return whatever is the current value:
>>> a = [lambda: i for i in range(5)]
>>> a[0]()
4
>>> i = 'foobar'
>>> a[0]()
'foobar'
Since the value of i within the loop is the lambda itself you’ll get it as a return value:
>>> i = a[0]
>>> i()
<function <lambda> at 0x01D689F0>
>>> i()()()()
<function <lambda> at 0x01D689F0>
UPDATE: Example on Python 2.7:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
... print i()
...
<function <lambda> at 0x7f1eae7f15f0>
<function <lambda> at 0x7f1eae7f1668>
<function <lambda> at 0x7f1eae7f16e0>
<function <lambda> at 0x7f1eae7f1758>
<function <lambda> at 0x7f1eae7f17d0>
Same on Python 3.4:
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
... print(i())
...
4
4
4
4
4
For details about the change regarding the variable scope with list comprehension see Guido’s blogpost from 2010.
We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension “leaks” the loop control variable into the surrounding scope:
x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'
However, in Python 3, we decided to fix the “dirty little secret” of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) 🙂 will print ‘before’, proving that the ‘x’ in the list comprehension temporarily shadows but does not override the ‘x’ in the surrounding scope.
I am trying to iterate the lambda func over a list as in test.py, and I want to get the call result of the lambda, not the function object itself. However, the following output really confused me.
------test.py---------
#!/bin/env python
#coding: utf-8
a = [lambda: i for i in range(5)]
for i in a:
print i()
--------output---------
<function <lambda> at 0x7f489e542e60>
<function <lambda> at 0x7f489e542ed8>
<function <lambda> at 0x7f489e542f50>
<function <lambda> at 0x7f489e54a050>
<function <lambda> at 0x7f489e54a0c8>
I modified the variable name when print the call result to t as following, and everything goes well. I am wondering what is all about of that. ?
--------test.py(update)--------
a = [lambda: i for i in range(5)]
for t in a:
print t()
-----------output-------------
4
4
4
4
4
This behaviour is 2.x-specific, and is a special case of the problem described at What do lambda function closures capture?. In 3.x, the list comprehension creates its own scope for the iteration variable, so it isn’t modifiable from outside. It is, however, still late binding, so each print will produce 4.
Closures in Python are late-binding, meaning that each lambda function in the list will only evaluate the variable i when invoked, and not when defined. That’s why all functions return the same value, i.e. the last value of ì (which is 4).
To avoid this, one technique is to bind the value of i to a local named parameter:
>>> a = [lambda i=i: i for i in range(5)]
>>> for t in a:
... print t()
...
0
1
2
3
4
Another option is to create a partial function and bind the current value of i as its parameter:
>>> from functools import partial
>>> a = [partial(lambda x: x, i) for i in range(5)]
>>> for t in a:
... print t()
...
0
1
2
3
4
Edit: Sorry, misread the question initially, since these kind of questions are so often about late binding (thanks @soon for the comment).
The second reason for the behavior is list comprehension’s variable leaking in Python2 as others have already explained. When using i as the iteration variable in the for loop, each function prints the current value of i (for the reasons stated above), which is simply the function itself.
When using a different name (e.g. t), functions print the last value of i as it was in the list comprehension loop, which is 4.
lambda: i is an anonymous function with no arguments that returns i. So you are generating a list of anonymous functions, which you can later (in the second example) bind to the name t and invoke with (). Note you can do the same with non-anonymous functions:
>>> def f():
... return 42
...
>>> name = f
>>> name
<function f at 0x7fed4d70fb90>
>>> name()
42
@plamut has just answered the implied other part of the question, so I won’t.
In Python 2 list comprehension ‘leaks’ the variables to outer scope:
>>> [i for i in xrange(3)]
[0, 1, 2]
>>> i
2
Note that the behavior is different on Python 3:
>>> [i for i in range(3)]
[0, 1, 2]
>>> i
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'i' is not defined
When you define lambda it’s bound to variable i, not its’ current value as your second example shows.
Now when you assign new value to i the lambda will return whatever is the current value:
>>> a = [lambda: i for i in range(5)]
>>> a[0]()
4
>>> i = 'foobar'
>>> a[0]()
'foobar'
Since the value of i within the loop is the lambda itself you’ll get it as a return value:
>>> i = a[0]
>>> i()
<function <lambda> at 0x01D689F0>
>>> i()()()()
<function <lambda> at 0x01D689F0>
UPDATE: Example on Python 2.7:
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
... print i()
...
<function <lambda> at 0x7f1eae7f15f0>
<function <lambda> at 0x7f1eae7f1668>
<function <lambda> at 0x7f1eae7f16e0>
<function <lambda> at 0x7f1eae7f1758>
<function <lambda> at 0x7f1eae7f17d0>
Same on Python 3.4:
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
... print(i())
...
4
4
4
4
4
For details about the change regarding the variable scope with list comprehension see Guido’s blogpost from 2010.
We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension “leaks” the loop control variable into the surrounding scope:
x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'
However, in Python 3, we decided to fix the “dirty little secret” of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) 🙂 will print ‘before’, proving that the ‘x’ in the list comprehension temporarily shadows but does not override the ‘x’ in the surrounding scope.