Strange result when removing item from a list while iterating over it
Question:
I’ve got this piece of code:
numbers = list(range(1, 50))
for i in numbers:
if i < 20:
numbers.remove(i)
print(numbers)
but the result I’m getting is:
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
Of course, I’m expecting the numbers below 20 to not appear in the results. Looks like I’m doing something wrong with the remove.
Answers:
You’re modifying the list while you iterate over it. That means that the first time through the loop, i == 1, so 1 is removed from the list. Then the for loop goes to the second item in the list, which is not 2, but 3! Then that’s removed from the list, and then the for loop goes on to the third item in the list, which is now 5. And so on. Perhaps it’s easier to visualize like so, with a ^ pointing to the value of i:
[1, 2, 3, 4, 5, 6...]
^
That’s the state of the list initially; then 1 is removed and the loop goes to the second item in the list:
[2, 3, 4, 5, 6...]
^
[2, 4, 5, 6...]
^
And so on.
There’s no good way to alter a list’s length while iterating over it. The best you can do is something like this:
numbers = [n for n in numbers if n >= 20]
or this, for in-place alteration (the thing in parens is a generator expression, which is implicitly converted into a tuple before slice-assignment):
numbers[:] = (n for in in numbers if n >= 20)
If you want to perform an operation on n before removing it, one trick you could try is this:
for i, n in enumerate(numbers):
if n < 20 :
print("do something")
numbers[i] = None
numbers = [n for n in numbers if n is not None]
@senderle’s answer is the way to go!
Having said that to further illustrate even a bit more your problem, if you think about it, you will always want to remove the index 0 twenty times:
[1,2,3,4,5............50]
^
[2,3,4,5............50]
^
[3,4,5............50]
^
So you could actually go with something like this:
aList = list(range(50))
i = 0
while i < 20:
aList.pop(0)
i += 1
print(aList) #[21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
I hope it helps.
The ones below are not bad practices AFAIK.
EDIT (Some more):
lis = range(50)
lis = lis[20:]
Will do the job also.
EDIT2 (I’m bored):
functional = filter(lambda x: x> 20, range(50))
Begin at the list’s end and go backwards:
li = list(range(1, 15))
print(li)
for i in range(len(li) - 1, -1, -1):
if li[i] < 6:
del li[i]
print(li)
Result:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
[6, 7, 8, 9, 10, 11, 12, 13, 14]
So I found a solution but it’s really clumsy…
First of all you make an index array, where you list all the index’ you want to delete like in the following
numbers = range(1, 50)
index_arr = []
for i in range(len(numbers):
if numbers[i] < 20:
index_arr.append(i)
after that you want to delete all the entries from the numbers list with the index saved in the index_arr. The problem you will encounter is the same as before. Therefore you have to subtract 1 from every index in the index_arr after you just removed a number from the numbers arr, like in the following:
numbers = range(1, 50)
index_arr = []
for i in range(len(numbers):
if numbers[i] < 20:
index_arr.append(i)
for del_index in index_list:
numbers.pop(del_index)
#the nasty part
for i in range(len(index_list)):
index_list[i] -= 1
It will work, but I guess it’s not the intended way to do it
As an additional information to @Senderle’s answer, just for records, I thought it’s helpful to visualize the logic behind the scene when python sees for on a "Sequence type".
Let’s say we have :
lst = [1, 2, 3, 4, 5]
for i in lst:
print(i ** 2)
It is actually going to be :
index = 0
while True:
try:
i = lst.__getitem__(index)
except IndexError:
break
print(i ** 2)
index += 1
That’s what it is, there is a try-catch mechanism that for has when we use it on a Sequence types or Iterables(It’s a little different though – calling next() and StopIteration Exception).
*All I’m trying to say is, python will keep track of an independent variable here called index, so no matter what happens to the list (removing or adding), python increments that variable and calls __getitem__() method with "this variable" and asks for item.
You could also use continue to ignore the values less than 20
mylist = []
for i in range(51):
if i<20:
continue
else:
mylist.append(i)
print(mylist)
Building on and simplying the answer by @eyquem …
The problem is that elements are being yanked out from under you as you iterate, skipping numbers as you progress to what was the next number.
If you start from the end and go backwards, removing items on-the-go won’t matter, because when it steps to the "next" item (actually the prior item), the deletion does not affect the first half of the list.
Simply adding reversed() to your iterator solves the problem. A comment would be good form to preclude future developers from "tidying up" your code and breaking it mysteriously.
for i in reversed(numbers): # `reversed` so removing doesn't foobar iteration
if i < 20:
numbers.remove(i)
Since Python 3.3 you may use the list copy() method as the iterator:
numbers = list(range(1, 50))
for i in numbers.copy():
if i < 20:
numbers.remove(i)
print(numbers)
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
I’ve got this piece of code:
numbers = list(range(1, 50))
for i in numbers:
if i < 20:
numbers.remove(i)
print(numbers)
but the result I’m getting is:
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
Of course, I’m expecting the numbers below 20 to not appear in the results. Looks like I’m doing something wrong with the remove.
You’re modifying the list while you iterate over it. That means that the first time through the loop, i == 1, so 1 is removed from the list. Then the for loop goes to the second item in the list, which is not 2, but 3! Then that’s removed from the list, and then the for loop goes on to the third item in the list, which is now 5. And so on. Perhaps it’s easier to visualize like so, with a ^ pointing to the value of i:
[1, 2, 3, 4, 5, 6...]
^
That’s the state of the list initially; then 1 is removed and the loop goes to the second item in the list:
[2, 3, 4, 5, 6...]
^
[2, 4, 5, 6...]
^
And so on.
There’s no good way to alter a list’s length while iterating over it. The best you can do is something like this:
numbers = [n for n in numbers if n >= 20]
or this, for in-place alteration (the thing in parens is a generator expression, which is implicitly converted into a tuple before slice-assignment):
numbers[:] = (n for in in numbers if n >= 20)
If you want to perform an operation on n before removing it, one trick you could try is this:
for i, n in enumerate(numbers):
if n < 20 :
print("do something")
numbers[i] = None
numbers = [n for n in numbers if n is not None]
@senderle’s answer is the way to go!
Having said that to further illustrate even a bit more your problem, if you think about it, you will always want to remove the index 0 twenty times:
[1,2,3,4,5............50]
^
[2,3,4,5............50]
^
[3,4,5............50]
^
So you could actually go with something like this:
aList = list(range(50))
i = 0
while i < 20:
aList.pop(0)
i += 1
print(aList) #[21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
I hope it helps.
The ones below are not bad practices AFAIK.
EDIT (Some more):
lis = range(50)
lis = lis[20:]
Will do the job also.
EDIT2 (I’m bored):
functional = filter(lambda x: x> 20, range(50))
Begin at the list’s end and go backwards:
li = list(range(1, 15))
print(li)
for i in range(len(li) - 1, -1, -1):
if li[i] < 6:
del li[i]
print(li)
Result:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
[6, 7, 8, 9, 10, 11, 12, 13, 14]
So I found a solution but it’s really clumsy…
First of all you make an index array, where you list all the index’ you want to delete like in the following
numbers = range(1, 50)
index_arr = []
for i in range(len(numbers):
if numbers[i] < 20:
index_arr.append(i)
after that you want to delete all the entries from the numbers list with the index saved in the index_arr. The problem you will encounter is the same as before. Therefore you have to subtract 1 from every index in the index_arr after you just removed a number from the numbers arr, like in the following:
numbers = range(1, 50)
index_arr = []
for i in range(len(numbers):
if numbers[i] < 20:
index_arr.append(i)
for del_index in index_list:
numbers.pop(del_index)
#the nasty part
for i in range(len(index_list)):
index_list[i] -= 1
It will work, but I guess it’s not the intended way to do it
As an additional information to @Senderle’s answer, just for records, I thought it’s helpful to visualize the logic behind the scene when python sees for on a "Sequence type".
Let’s say we have :
lst = [1, 2, 3, 4, 5]
for i in lst:
print(i ** 2)
It is actually going to be :
index = 0
while True:
try:
i = lst.__getitem__(index)
except IndexError:
break
print(i ** 2)
index += 1
That’s what it is, there is a try-catch mechanism that for has when we use it on a Sequence types or Iterables(It’s a little different though – calling next() and StopIteration Exception).
*All I’m trying to say is, python will keep track of an independent variable here called index, so no matter what happens to the list (removing or adding), python increments that variable and calls __getitem__() method with "this variable" and asks for item.
You could also use continue to ignore the values less than 20
mylist = []
for i in range(51):
if i<20:
continue
else:
mylist.append(i)
print(mylist)
Building on and simplying the answer by @eyquem …
The problem is that elements are being yanked out from under you as you iterate, skipping numbers as you progress to what was the next number.
If you start from the end and go backwards, removing items on-the-go won’t matter, because when it steps to the "next" item (actually the prior item), the deletion does not affect the first half of the list.
Simply adding reversed() to your iterator solves the problem. A comment would be good form to preclude future developers from "tidying up" your code and breaking it mysteriously.
for i in reversed(numbers): # `reversed` so removing doesn't foobar iteration
if i < 20:
numbers.remove(i)
Since Python 3.3 you may use the list copy() method as the iterator:
numbers = list(range(1, 50))
for i in numbers.copy():
if i < 20:
numbers.remove(i)
print(numbers)
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]