List comprehension rebinds names even after scope of comprehension. Is this right?
Question:
Comprehensions are having some unexpected interactions with scoping. Is this the expected behavior?
I’ve got a method:
def leave_room(self, uid):
u = self.user_by_id(uid)
r = self.rooms[u.rid]
other_uids = [ouid for ouid in r.users_by_id.keys() if ouid != u.uid]
other_us = [self.user_by_id(uid) for uid in other_uids]
r.remove_user(uid) # OOPS! uid has been re-bound by the list comprehension above
# Interestingly, it's rebound to the last uid in the list, so the error only shows
# up when len > 1
At the risk of whining, this is a brutal source of errors. As I write new code, I just occasionally find very weird errors due to rebinding — even now that I know it’s a problem. I need to make a rule like “always preface temp vars in list comprehensions with underscore”, but even that’s not fool-proof.
The fact that there’s this random time-bomb waiting kind of negates all the nice “ease of use” of list comprehensions.
Answers:
Yes, assignment occurs there, just like it would in a for loop. No new scope is being created.
This is definitely the expected behavior: on each cycle, the value is bound to the name you specify. For instance,
>>> x=0
>>> a=[1,54,4,2,32,234,5234,]
>>> [x for x in a if x>32]
[54, 234, 5234]
>>> x
5234
Once that’s recognized, it seems easy enough to avoid: don’t use existing names for the variables within comprehensions.
Yes, list comprehensions “leak” their variable in Python 2.x, just like for loops.
In retrospect, this was recognized to be a mistake, and it was avoided with generator expressions. EDIT: As Matt B. notes it was also avoided when set and dictionary comprehension syntaxes were backported from Python 3.
List comprehensions’ behavior had to be left as it is in Python 2, but it’s fully fixed in Python 3.
This means that in all of:
list(x for x in a if x>32)
set(x//4 for x in a if x>32) # just another generator exp.
dict((x, x//16) for x in a if x>32) # yet another generator exp.
{x//4 for x in a if x>32} # 2.7+ syntax
{x: x//16 for x in a if x>32} # 2.7+ syntax
the x is always local to the expression while these:
[x for x in a if x>32]
set([x//4 for x in a if x>32]) # just another list comp.
dict([(x, x//16) for x in a if x>32]) # yet another list comp.
in Python 2.x all leak the x variable to the surrounding scope.
UPDATE for Python 3.8(?): PEP 572 will introduce := assignment operator that deliberately leaks out of comprehensions and generator expressions! It’s motivated by essentially 2 use cases: capturing a “witness” from early-terminating functions like any() and all():
if any((comment := line).startswith('#') for line in lines):
print("First comment:", comment)
else:
print("There are no comments")
and updating mutable state:
total = 0
partial_sums = [total := total + v for v in values]
See Appendix B for exact scoping. The variable is assigned in closest surrounding def or lambda, unless that function declares it nonlocal or global.
List comprehensions leak the loop control variable in Python 2 but not in Python 3. Here’s Guido van Rossum (creator of Python) explaining the history behind this:
We also made another change in Python
3, to improve equivalence between list
comprehensions and generator
expressions. In Python 2, the list
comprehension “leaks” the loop control
variable into the surrounding scope:
x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'
This was an artifact of the original
implementation of list comprehensions;
it was one of Python’s “dirty little
secrets” for years. It started out as
an intentional compromise to make list
comprehensions blindingly fast, and
while it was not a common pitfall for
beginners, it definitely stung people
occasionally. For generator
expressions we could not do this.
Generator expressions are implemented
using generators, whose execution
requires a separate execution frame.
Thus, generator expressions
(especially if they iterate over a
short sequence) were less efficient
than list comprehensions.
However, in Python 3, we decided to
fix the “dirty little secret” of list
comprehensions by using the same
implementation strategy as for
generator expressions. Thus, in Python
3, the above example (after
modification to use print(x) 🙂 will
print ‘before’, proving that the ‘x’
in the list comprehension temporarily
shadows but does not override the ‘x’
in the surrounding scope.
Interestingly this doesn’t affect dictionary or set comprehensions.
>>> [x for x in range(1, 10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> {x for x in range(1, 5)}
set([1, 2, 3, 4])
>>> x
9
>>> {x:x for x in range(1, 100)}
{1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99}
>>> x
9
However it has been fixed in 3 as noted above.
some workaround, for python 2.6, when this behaviour is not desirable
# python
Python 2.6.6 (r266:84292, Aug 9 2016, 06:11:56)
Type "help", "copyright", "credits" or "license" for more information.
>>> x=0
>>> a=list(x for x in xrange(9))
>>> x
0
>>> a=[x for x in xrange(9)]
>>> x
8
In python3 while in list comprehension the variable is not getting change after it’s scope over but when we use simple for-loop the variable is getting reassigned out of scope.
i = 1
print(i)
print([i in range(5)])
print(i)
Value of i will remain 1 only.
Now just use simply for loop the value of i will be reassigned.
Comprehensions are having some unexpected interactions with scoping. Is this the expected behavior?
I’ve got a method:
def leave_room(self, uid):
u = self.user_by_id(uid)
r = self.rooms[u.rid]
other_uids = [ouid for ouid in r.users_by_id.keys() if ouid != u.uid]
other_us = [self.user_by_id(uid) for uid in other_uids]
r.remove_user(uid) # OOPS! uid has been re-bound by the list comprehension above
# Interestingly, it's rebound to the last uid in the list, so the error only shows
# up when len > 1
At the risk of whining, this is a brutal source of errors. As I write new code, I just occasionally find very weird errors due to rebinding — even now that I know it’s a problem. I need to make a rule like “always preface temp vars in list comprehensions with underscore”, but even that’s not fool-proof.
The fact that there’s this random time-bomb waiting kind of negates all the nice “ease of use” of list comprehensions.
Yes, assignment occurs there, just like it would in a for loop. No new scope is being created.
This is definitely the expected behavior: on each cycle, the value is bound to the name you specify. For instance,
>>> x=0
>>> a=[1,54,4,2,32,234,5234,]
>>> [x for x in a if x>32]
[54, 234, 5234]
>>> x
5234
Once that’s recognized, it seems easy enough to avoid: don’t use existing names for the variables within comprehensions.
Yes, list comprehensions “leak” their variable in Python 2.x, just like for loops.
In retrospect, this was recognized to be a mistake, and it was avoided with generator expressions. EDIT: As Matt B. notes it was also avoided when set and dictionary comprehension syntaxes were backported from Python 3.
List comprehensions’ behavior had to be left as it is in Python 2, but it’s fully fixed in Python 3.
This means that in all of:
list(x for x in a if x>32)
set(x//4 for x in a if x>32) # just another generator exp.
dict((x, x//16) for x in a if x>32) # yet another generator exp.
{x//4 for x in a if x>32} # 2.7+ syntax
{x: x//16 for x in a if x>32} # 2.7+ syntax
the x is always local to the expression while these:
[x for x in a if x>32]
set([x//4 for x in a if x>32]) # just another list comp.
dict([(x, x//16) for x in a if x>32]) # yet another list comp.
in Python 2.x all leak the x variable to the surrounding scope.
UPDATE for Python 3.8(?): PEP 572 will introduce := assignment operator that deliberately leaks out of comprehensions and generator expressions! It’s motivated by essentially 2 use cases: capturing a “witness” from early-terminating functions like any() and all():
if any((comment := line).startswith('#') for line in lines):
print("First comment:", comment)
else:
print("There are no comments")
and updating mutable state:
total = 0
partial_sums = [total := total + v for v in values]
See Appendix B for exact scoping. The variable is assigned in closest surrounding def or lambda, unless that function declares it nonlocal or global.
List comprehensions leak the loop control variable in Python 2 but not in Python 3. Here’s Guido van Rossum (creator of Python) explaining the history behind this:
We also made another change in Python
3, to improve equivalence between list
comprehensions and generator
expressions. In Python 2, the list
comprehension “leaks” the loop control
variable into the surrounding scope:x = 'before' a = [x for x in 1, 2, 3] print x # this prints '3', not 'before'This was an artifact of the original
implementation of list comprehensions;
it was one of Python’s “dirty little
secrets” for years. It started out as
an intentional compromise to make list
comprehensions blindingly fast, and
while it was not a common pitfall for
beginners, it definitely stung people
occasionally. For generator
expressions we could not do this.
Generator expressions are implemented
using generators, whose execution
requires a separate execution frame.
Thus, generator expressions
(especially if they iterate over a
short sequence) were less efficient
than list comprehensions.However, in Python 3, we decided to
fix the “dirty little secret” of list
comprehensions by using the same
implementation strategy as for
generator expressions. Thus, in Python
3, the above example (after
modification to use print(x) 🙂 will
print ‘before’, proving that the ‘x’
in the list comprehension temporarily
shadows but does not override the ‘x’
in the surrounding scope.
Interestingly this doesn’t affect dictionary or set comprehensions.
>>> [x for x in range(1, 10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> {x for x in range(1, 5)}
set([1, 2, 3, 4])
>>> x
9
>>> {x:x for x in range(1, 100)}
{1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99}
>>> x
9
However it has been fixed in 3 as noted above.
some workaround, for python 2.6, when this behaviour is not desirable
# python
Python 2.6.6 (r266:84292, Aug 9 2016, 06:11:56)
Type "help", "copyright", "credits" or "license" for more information.
>>> x=0
>>> a=list(x for x in xrange(9))
>>> x
0
>>> a=[x for x in xrange(9)]
>>> x
8
In python3 while in list comprehension the variable is not getting change after it’s scope over but when we use simple for-loop the variable is getting reassigned out of scope.
i = 1
print(i)
print([i in range(5)])
print(i)
Value of i will remain 1 only.
Now just use simply for loop the value of i will be reassigned.