Why does Python skip elements when I modify a list while iterating over it?
Question:
I’m currently developing a program in python and I just noticed that something was wrong with the foreach loop in the language, or maybe the list structure. I’ll just give a generic example of my problem to simplify, since I get the same erroneous behavior on both my program and my generic example:
x = [1,2,2,2,2]
for i in x:
x.remove(i)
print x
Well, the problem here is simple, I though that this code was supposed to remove all elements from a list. Well, the problem is that after it’s execution, I always get 2 remaining elements in the list.
What am I doing wrong? Thanks for all the help in advance.
Edit: I don’t want to empty a list, this is just an example…
Answers:
This is a well-documented behaviour in Python, that you aren’t supposed to modify the list being iterated through. Try this instead:
for i in x[:]:
x.remove(i)
The [:] returns a “slice” of x, which happens to contain all its elements, and is thus effectively a copy of x.
When you delete an element, and the for-loop incs to the next index, you then skip an element.
Do it backwards. Or please state your real problem.
Why don’t you just use:
x = []
It’s probably because you’re changing the same array that you’re iterating over.
Try Chris-Jester Young’s answer if you want to clear the array your way.
I think, broadly speaking, that when you write:
for x in lst:
# loop body goes here
under the hood, python is doing something like this:
i = 0
while i < len(lst):
x = lst[i]
# loop body goes here
i += 1
If you insert lst.remove(x) for the loop body, perhaps then you’ll be able to see why you get the result you do?
Essentially, python uses a moving pointer to traverse the list. The pointer starts by pointing at the first element. Then you remove the first element, thus making the second element the new first element. Then the pointer move to the new second – previously third – element. And so on. (it might be clearer if you use [1,2,3,4,5] instead of [1,2,2,2,2] as your sample list)
I agree with John Fouhy regarding the break condition. Traversing a copy of the list works for the remove() method, as Chris Jester-Young suggested. But if one needs to pop() specific items, then iterating in reverse works, as Erik mentioned, in which case the operation can be done in place. For example:
def r_enumerate(iterable):
"""enumerator for reverse iteration of an iterable"""
enum = enumerate(reversed(iterable))
last = len(iterable)-1
return ((last - i, x) for i,x in enum)
x = [1,2,3,4,5]
y = []
for i,v in r_enumerate(x):
if v != 3:
y.append(x.pop(i))
print 'i=%d, v=%d, x=%s, y=%s' %(i,v,x,y)
or with xrange:
x = [1,2,3,4,5]
y = []
for i in xrange(len(x)-1,-1,-1):
if x[i] != 3:
y.append(x.pop(i))
print 'i=%d, x=%s, y=%s' %(i,x,y)
I know this is an old post with an accepted answer but for those that may still come along…
A few previous answers have indicated it’s a bad idea to change an iterable during iteration. But as a way to highlight what is happening…
>>> x=[1,2,3,4,5]
>>> for i in x:
... print i, x.index(i)
... x.remove(i)
... print x
...
1 0
[2, 3, 4, 5]
3 1
[2, 4, 5]
5 2
[2, 4]
Hopefully the visual helps clarify.
If you need to filter stuff out of a list it may be a better idea to use list comprehension:
newlist = [x for x in oldlist if x%2]
for instance would filter all even numbers out of an integer list
The list stored in the memory of a computer. This deals with the pointer to a memory artifact. When you remove an element, in a by-element loop, you are then moving the pointer to the next available element in the memory address
You are modifying the memory and iterating thru the same.
The pointer to the element moves through the list to the next spot available.
So in the case of the Size being 5…enter code here
[**0**,1,2,3,4]
remove 0 ---> [1,**2**,3,4] pointer moves to second index.
remove 2 ---> [1,3,**4**] pointer moves to 3rd index.
remove 4 ---> [1,3]
I was just explaining this to my students when they used pop(1). Another very interesting side-effect error.
x=[1,**2**,3,4,5]
for i in x:
x.pop(1)
print(x,i)
[1, **3**, 4, 5] 1 at index 0 it removed the index 1 (2)
[1, **4**, 5] 3 at index 1 it removed the index 1 (3)
[1, 5] 5 at index 2 it removed the index 1 (4)
heh.
They were like why isnt this working… I mean… it did… exactly what you told it to do. Not a mind reader. 🙂
I’m currently developing a program in python and I just noticed that something was wrong with the foreach loop in the language, or maybe the list structure. I’ll just give a generic example of my problem to simplify, since I get the same erroneous behavior on both my program and my generic example:
x = [1,2,2,2,2]
for i in x:
x.remove(i)
print x
Well, the problem here is simple, I though that this code was supposed to remove all elements from a list. Well, the problem is that after it’s execution, I always get 2 remaining elements in the list.
What am I doing wrong? Thanks for all the help in advance.
Edit: I don’t want to empty a list, this is just an example…
This is a well-documented behaviour in Python, that you aren’t supposed to modify the list being iterated through. Try this instead:
for i in x[:]:
x.remove(i)
The [:] returns a “slice” of x, which happens to contain all its elements, and is thus effectively a copy of x.
When you delete an element, and the for-loop incs to the next index, you then skip an element.
Do it backwards. Or please state your real problem.
Why don’t you just use:
x = []
It’s probably because you’re changing the same array that you’re iterating over.
Try Chris-Jester Young’s answer if you want to clear the array your way.
I think, broadly speaking, that when you write:
for x in lst:
# loop body goes here
under the hood, python is doing something like this:
i = 0
while i < len(lst):
x = lst[i]
# loop body goes here
i += 1
If you insert lst.remove(x) for the loop body, perhaps then you’ll be able to see why you get the result you do?
Essentially, python uses a moving pointer to traverse the list. The pointer starts by pointing at the first element. Then you remove the first element, thus making the second element the new first element. Then the pointer move to the new second – previously third – element. And so on. (it might be clearer if you use [1,2,3,4,5] instead of [1,2,2,2,2] as your sample list)
I agree with John Fouhy regarding the break condition. Traversing a copy of the list works for the remove() method, as Chris Jester-Young suggested. But if one needs to pop() specific items, then iterating in reverse works, as Erik mentioned, in which case the operation can be done in place. For example:
def r_enumerate(iterable):
"""enumerator for reverse iteration of an iterable"""
enum = enumerate(reversed(iterable))
last = len(iterable)-1
return ((last - i, x) for i,x in enum)
x = [1,2,3,4,5]
y = []
for i,v in r_enumerate(x):
if v != 3:
y.append(x.pop(i))
print 'i=%d, v=%d, x=%s, y=%s' %(i,v,x,y)
or with xrange:
x = [1,2,3,4,5]
y = []
for i in xrange(len(x)-1,-1,-1):
if x[i] != 3:
y.append(x.pop(i))
print 'i=%d, x=%s, y=%s' %(i,x,y)
I know this is an old post with an accepted answer but for those that may still come along…
A few previous answers have indicated it’s a bad idea to change an iterable during iteration. But as a way to highlight what is happening…
>>> x=[1,2,3,4,5]
>>> for i in x:
... print i, x.index(i)
... x.remove(i)
... print x
...
1 0
[2, 3, 4, 5]
3 1
[2, 4, 5]
5 2
[2, 4]
Hopefully the visual helps clarify.
If you need to filter stuff out of a list it may be a better idea to use list comprehension:
newlist = [x for x in oldlist if x%2]
for instance would filter all even numbers out of an integer list
The list stored in the memory of a computer. This deals with the pointer to a memory artifact. When you remove an element, in a by-element loop, you are then moving the pointer to the next available element in the memory address
You are modifying the memory and iterating thru the same.
The pointer to the element moves through the list to the next spot available.
So in the case of the Size being 5…enter code here
[**0**,1,2,3,4]
remove 0 ---> [1,**2**,3,4] pointer moves to second index.
remove 2 ---> [1,3,**4**] pointer moves to 3rd index.
remove 4 ---> [1,3]
I was just explaining this to my students when they used pop(1). Another very interesting side-effect error.
x=[1,**2**,3,4,5]
for i in x:
x.pop(1)
print(x,i)
[1, **3**, 4, 5] 1 at index 0 it removed the index 1 (2)
[1, **4**, 5] 3 at index 1 it removed the index 1 (3)
[1, 5] 5 at index 2 it removed the index 1 (4)
heh.
They were like why isnt this working… I mean… it did… exactly what you told it to do. Not a mind reader. 🙂