How are deques in Python implemented, and when are they worse than lists?
Question:
I’ve recently gotten into investigating how various data structures are implemented in Python in order to make my code more efficient. In investigating how lists and deques work, I found that I can get benefits when I want to shift and unshift reducing the time from O(n) in lists to O(1) in deques (lists being implemented as fixed-length arrays that have to be copied completely each time something is inserted at the front, etc…). What I can’t seem to find are the specifics of how a deque is implemented, and the specifics of its downsides v.s. lists. Can someone enlighten me on these two questions?
Answers:
Check out collections.deque. From the docs:
Deques support thread-safe, memory
efficient appends and pops from either
side of the deque with approximately
the same O(1) performance in either
direction.
Though list objects support similar
operations, they are optimized for
fast fixed-length operations and incur
O(n) memory movement costs for pop(0)
and insert(0, v) operations which
change both the size and position of
the underlying data representation.
Just as it says, using pop(0) or insert(0, v) incur large penalties with list objects. You can’t use slice/index operations on a deque, but you can use popleft/appendleft, which are operations deque is optimized for. Here is a simple benchmark to demonstrate this:
import time
from collections import deque
num = 100000
def append(c):
for i in range(num):
c.append(i)
def appendleft(c):
if isinstance(c, deque):
for i in range(num):
c.appendleft(i)
else:
for i in range(num):
c.insert(0, i)
def pop(c):
for i in range(num):
c.pop()
def popleft(c):
if isinstance(c, deque):
for i in range(num):
c.popleft()
else:
for i in range(num):
c.pop(0)
for container in [deque, list]:
for operation in [append, appendleft, pop, popleft]:
c = container(range(num))
start = time.time()
operation(c)
elapsed = time.time() - start
print "Completed %s/%s in %.2f seconds: %.1f ops/sec" % (container.__name__, operation.__name__, elapsed, num / elapsed)
Results on my machine:
Completed deque/append in 0.02 seconds: 5582877.2 ops/sec
Completed deque/appendleft in 0.02 seconds: 6406549.7 ops/sec
Completed deque/pop in 0.01 seconds: 7146417.7 ops/sec
Completed deque/popleft in 0.01 seconds: 7271174.0 ops/sec
Completed list/append in 0.01 seconds: 6761407.6 ops/sec
Completed list/appendleft in 16.55 seconds: 6042.7 ops/sec
Completed list/pop in 0.02 seconds: 4394057.9 ops/sec
Completed list/popleft in 3.23 seconds: 30983.3 ops/sec
https://github.com/python/cpython/blob/v3.8.1/Modules/_collectionsmodule.c
A dequeobject is composed of a doubly-linked list of block nodes.
So yes, a deque is a (doubly-)linked list as another answer suggests.
Elaborating: What this means is that Python lists are much better for random-access and fixed-length operations, including slicing, while deques are much more useful for pushing and popping things off the ends, with indexing (but not slicing, interestingly) being possible but slower than with lists.
The documentation entry for deque objects spells out most of what you need to know, I suspect. Notable quotes:
Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.
But…
Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead.
I’d have to take a look at the source to tell whether the implementation is a linked list or something else, but it sounds to me as though a deque has roughly the same characteristics as a doubly-linked list.
In addition to all the other helpful answers, here is some more information comparing the time complexity (Big-Oh) of various operations on Python lists, deques, sets, and dictionaries. This should help in selecting the right data structure for a particular problem.
I’ve recently gotten into investigating how various data structures are implemented in Python in order to make my code more efficient. In investigating how lists and deques work, I found that I can get benefits when I want to shift and unshift reducing the time from O(n) in lists to O(1) in deques (lists being implemented as fixed-length arrays that have to be copied completely each time something is inserted at the front, etc…). What I can’t seem to find are the specifics of how a deque is implemented, and the specifics of its downsides v.s. lists. Can someone enlighten me on these two questions?
Check out collections.deque. From the docs:
Deques support thread-safe, memory
efficient appends and pops from either
side of the deque with approximately
the same O(1) performance in either
direction.Though list objects support similar
operations, they are optimized for
fast fixed-length operations and incur
O(n) memory movement costs for pop(0)
and insert(0, v) operations which
change both the size and position of
the underlying data representation.
Just as it says, using pop(0) or insert(0, v) incur large penalties with list objects. You can’t use slice/index operations on a deque, but you can use popleft/appendleft, which are operations deque is optimized for. Here is a simple benchmark to demonstrate this:
import time
from collections import deque
num = 100000
def append(c):
for i in range(num):
c.append(i)
def appendleft(c):
if isinstance(c, deque):
for i in range(num):
c.appendleft(i)
else:
for i in range(num):
c.insert(0, i)
def pop(c):
for i in range(num):
c.pop()
def popleft(c):
if isinstance(c, deque):
for i in range(num):
c.popleft()
else:
for i in range(num):
c.pop(0)
for container in [deque, list]:
for operation in [append, appendleft, pop, popleft]:
c = container(range(num))
start = time.time()
operation(c)
elapsed = time.time() - start
print "Completed %s/%s in %.2f seconds: %.1f ops/sec" % (container.__name__, operation.__name__, elapsed, num / elapsed)
Results on my machine:
Completed deque/append in 0.02 seconds: 5582877.2 ops/sec
Completed deque/appendleft in 0.02 seconds: 6406549.7 ops/sec
Completed deque/pop in 0.01 seconds: 7146417.7 ops/sec
Completed deque/popleft in 0.01 seconds: 7271174.0 ops/sec
Completed list/append in 0.01 seconds: 6761407.6 ops/sec
Completed list/appendleft in 16.55 seconds: 6042.7 ops/sec
Completed list/pop in 0.02 seconds: 4394057.9 ops/sec
Completed list/popleft in 3.23 seconds: 30983.3 ops/sec
https://github.com/python/cpython/blob/v3.8.1/Modules/_collectionsmodule.c
A
dequeobjectis composed of a doubly-linked list ofblocknodes.
So yes, a deque is a (doubly-)linked list as another answer suggests.
Elaborating: What this means is that Python lists are much better for random-access and fixed-length operations, including slicing, while deques are much more useful for pushing and popping things off the ends, with indexing (but not slicing, interestingly) being possible but slower than with lists.
The documentation entry for deque objects spells out most of what you need to know, I suspect. Notable quotes:
Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction.
But…
Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead.
I’d have to take a look at the source to tell whether the implementation is a linked list or something else, but it sounds to me as though a deque has roughly the same characteristics as a doubly-linked list.
In addition to all the other helpful answers, here is some more information comparing the time complexity (Big-Oh) of various operations on Python lists, deques, sets, and dictionaries. This should help in selecting the right data structure for a particular problem.