Best way to remove (not pop) last/rightmost object occurrence from a stack using collections.deque
Question:
I am using the standard collections.deque to write a LIFO stack where each object may occur multiple times, but now I am cornered around the use case for removing the last occurrence of a given object (but not whatever is the rightmost object of the stack!).
While appendleft, extendleft and popleft counterparts exist for these three methods, no removeright (nor indexright) exist. So the following is not possible.
import collections
stack = collections.deque()
a = object()
b = object()
c = object()
stack.append(a)
stack.append(b)
stack.append(c)
stack.append(a)
stack.append(b)
stack.append(c)
list(stack) # [a, b, c, a, b, c]
stack.removeright(b) # Fat chance
list(stack) # Whish: [a, b, c, a, c] and *NOT* [a, c, a, b, c]
Am I missing something obvious?
Right now I am going with a double reverse call like
def removeright(stack, item):
stack.reverse()
try:
stack.remove(item)
finally:
stack.reverse()
but this feels wrong. I am worried about both inefficiency and potential pitfalls down the road for this approach.
I could always use the queue "backwards" (actually quite conventional), using appendleft and remove, but I’d like to retain the "append" semantics and still not have to write a thin wrapper patching every right/left stack method to a left/right queue method.
Would someone share their insights/experience on the subject?
Answers:
The only other way I could find would be to start from the last position and decrement until you find the item you’re looking for.
def for_remove(stack, s):
# Optionally check if the element is in the stack first
if s not in stack:
return
for i in range(len(stack)-1, -1, -1):
if s == stack[i]:
del stack[i]
break
Although this avoids two reversal operations, you are still having to loop through your deque in Python. Although clunky, your solution has the advantage of using speed-optimised, compiled, methods and is most likely faster than the above solution.
The question is: is this level of optimisation warranted for your current task? Do you need to be shaving off microseconds from the deletion time? In most cases, your containers will be small enough that any optimizations have a negligble impact on the runtime.
As I can’t come up with a simple solution to this problem, I now believe a little semantics sacrifice is worth it and just use the collections.deque object "backwards".
Simply put: consider the "left" end to be the "top" of the stack.
Bonus 1: when not popping the stack iterates naturally in a LIFO order
Bonus 2: for the probable case when the removed object is close to the last position (now "leftmost"), CPython’s deque.remove will perform the smallest rotation of the internal datastructure. Up to no operation at all when removing the very last item. So this should be blazing fast. Screw lists!
import collections
stack = collections.deque()
stack.appendleft("a")
stack.appendleft("b")
stack.appendleft("c")
stack.appendleft("a")
stack.appendleft("b")
stack.appendleft("c")
list(stack) # [c, b, a, c, b, a]
stack.remove("b")
def consume():
while stack:
yield stack.popleft()
list(consume()) # yields [c, a, c, b, a]
Drawback: the issue is reverted as there is no simple formula for removal of the first (now "rightmost") occurrence of a given object in the deque. Therefore, using the deque objects with this "reversed semantics" should be conditioned to whether your removing/indexing use-case is for either the last or the first occurrence of a stack.
For a general solution, keep reading.
If you are really into removing the rightmost occurrence in a deque object.
It turns out deque.remove is internally implemented with a double rotation until the desired item is at an end, when it is popped and then the deque object is rotated back into the original state. This is a CPython implementation detail, though. See
- https://github.com/python/cpython/blob/ee821dcd3961efc47262322848267fe398faa4e4/Modules/_collectionsmodule.c#L1249
- https://github.com/python/cpython/blob/ee821dcd3961efc47262322848267fe398faa4e4/Modules/_collectionsmodule.c#L1200
So a consistent and efficient solution might be a matter of iteratively finding the last index of a multiply present object and also doing a double rotation instead of a double reversion.
For my use case, the item to be removed will quite often be the actual last item of the whole stack. So the following implementation favors rotating as few positions as possible, up to the corner case of rotating none and only popping from the right.
Also, it is nice to retain the ValueError being raised in absence of the item, thus the unavoidable deque.index call.
def index_right(stack, item):
idx = stack.index(item)
for _ in range(1, stack.count(item)):
idx = stack.index(item, idx + 1)
return idx
def remove_right(stack, item):
rot = len(stack) - index_right(stack, item) - 1
d.rotate(rot)
try:
d.pop()
finally:
d.rotate(-rot)
A drawback is: for a big stack this may take a lot of unnecessary value comparisons because deque.index operates from left to right and the searched items will usually be at the end. So, the revert-remove-revert solution from the original question might still outperform this answer under certain conditions.
I am using the standard collections.deque to write a LIFO stack where each object may occur multiple times, but now I am cornered around the use case for removing the last occurrence of a given object (but not whatever is the rightmost object of the stack!).
While appendleft, extendleft and popleft counterparts exist for these three methods, no removeright (nor indexright) exist. So the following is not possible.
import collections
stack = collections.deque()
a = object()
b = object()
c = object()
stack.append(a)
stack.append(b)
stack.append(c)
stack.append(a)
stack.append(b)
stack.append(c)
list(stack) # [a, b, c, a, b, c]
stack.removeright(b) # Fat chance
list(stack) # Whish: [a, b, c, a, c] and *NOT* [a, c, a, b, c]
Am I missing something obvious?
Right now I am going with a double reverse call like
def removeright(stack, item):
stack.reverse()
try:
stack.remove(item)
finally:
stack.reverse()
but this feels wrong. I am worried about both inefficiency and potential pitfalls down the road for this approach.
I could always use the queue "backwards" (actually quite conventional), using appendleft and remove, but I’d like to retain the "append" semantics and still not have to write a thin wrapper patching every right/left stack method to a left/right queue method.
Would someone share their insights/experience on the subject?
The only other way I could find would be to start from the last position and decrement until you find the item you’re looking for.
def for_remove(stack, s):
# Optionally check if the element is in the stack first
if s not in stack:
return
for i in range(len(stack)-1, -1, -1):
if s == stack[i]:
del stack[i]
break
Although this avoids two reversal operations, you are still having to loop through your deque in Python. Although clunky, your solution has the advantage of using speed-optimised, compiled, methods and is most likely faster than the above solution.
The question is: is this level of optimisation warranted for your current task? Do you need to be shaving off microseconds from the deletion time? In most cases, your containers will be small enough that any optimizations have a negligble impact on the runtime.
As I can’t come up with a simple solution to this problem, I now believe a little semantics sacrifice is worth it and just use the collections.deque object "backwards".
Simply put: consider the "left" end to be the "top" of the stack.
Bonus 1: when not popping the stack iterates naturally in a LIFO order
Bonus 2: for the probable case when the removed object is close to the last position (now "leftmost"), CPython’s deque.remove will perform the smallest rotation of the internal datastructure. Up to no operation at all when removing the very last item. So this should be blazing fast. Screw lists!
import collections
stack = collections.deque()
stack.appendleft("a")
stack.appendleft("b")
stack.appendleft("c")
stack.appendleft("a")
stack.appendleft("b")
stack.appendleft("c")
list(stack) # [c, b, a, c, b, a]
stack.remove("b")
def consume():
while stack:
yield stack.popleft()
list(consume()) # yields [c, a, c, b, a]
Drawback: the issue is reverted as there is no simple formula for removal of the first (now "rightmost") occurrence of a given object in the deque. Therefore, using the deque objects with this "reversed semantics" should be conditioned to whether your removing/indexing use-case is for either the last or the first occurrence of a stack.
For a general solution, keep reading.
If you are really into removing the rightmost occurrence in a deque object.
It turns out deque.remove is internally implemented with a double rotation until the desired item is at an end, when it is popped and then the deque object is rotated back into the original state. This is a CPython implementation detail, though. See
- https://github.com/python/cpython/blob/ee821dcd3961efc47262322848267fe398faa4e4/Modules/_collectionsmodule.c#L1249
- https://github.com/python/cpython/blob/ee821dcd3961efc47262322848267fe398faa4e4/Modules/_collectionsmodule.c#L1200
So a consistent and efficient solution might be a matter of iteratively finding the last index of a multiply present object and also doing a double rotation instead of a double reversion.
For my use case, the item to be removed will quite often be the actual last item of the whole stack. So the following implementation favors rotating as few positions as possible, up to the corner case of rotating none and only popping from the right.
Also, it is nice to retain the ValueError being raised in absence of the item, thus the unavoidable deque.index call.
def index_right(stack, item):
idx = stack.index(item)
for _ in range(1, stack.count(item)):
idx = stack.index(item, idx + 1)
return idx
def remove_right(stack, item):
rot = len(stack) - index_right(stack, item) - 1
d.rotate(rot)
try:
d.pop()
finally:
d.rotate(-rot)
A drawback is: for a big stack this may take a lot of unnecessary value comparisons because deque.index operates from left to right and the searched items will usually be at the end. So, the revert-remove-revert solution from the original question might still outperform this answer under certain conditions.