What is the time complexity of heapq.nlargest?
Question:
I was looking at this pycon talk, 34:30 and the speaker says that getting the t largest elements of a list of n elements can be done in O(t + n).
How is that possible? My understanding is that creating the heap will be O(n), but what’s the complexity of nlargest itself, is it O(n + t) or O(t) (and what’s the actual algorithm)?
Answers:
The speaker is wrong in this case. The actual cost is O(n * log(t)). Heapify is called only on the first t elements of the iterable. That’s O(t), but is insignificant if t is much smaller than n. Then all the remaining elements are added to this “little heap” via heappushpop, one at a time. That takes O(log(t)) time per invocation of heappushpop. The length of the heap remains t throughout. At the very end, the heap is sorted, which costs O(t * log(t)), but that’s also insignificant if t is much smaller than n.
Fun with Theory 😉
There are reasonably easy ways to find the t’th-largest element in expected O(n) time; for example, see here. There are harder ways to do it in worst-case O(n) time. Then, in another pass over the input, you could output the t elements >= the t-th largest (with tedious complications in case of duplicates). So the whole job can be done in O(n) time.
But those ways require O(n) memory too. Python doesn’t use them. An advantage of what’s actually implemented is that the worst-case “extra” memory burden is O(t), and that can be very significant when the input is, for example, a generator producing a great many values.
It’s actually O(n+tlog(n)) because heapify takes O(n) and for every element of largest or smallest takes O(log(n)). So for t largest/smallest it takes tlog(n). Therefore time complexity will be O(n+t*log(n))
For Heapq t largest or t smallest, the time complexity will be O(nlog(t))
Heapq will build the heap for the first t elements, then later on it will iterate over the remaining elements by pushing and popping the elements from the heap (maintaining the t elements in the heap).
- For building the heap for first t elements will be done
tlog(t)
- For pushing and popping, the remaining elements will be done in
(n-t)log(t)
- The overall time complexity will be
nlog(t)
According to the comments in the heapq source code:
# Algorithm notes for nlargest() and nsmallest()
# ==============================================
#
# Make a single pass over the data while keeping the k most extreme values
# in a heap. Memory consumption is limited to keeping k values in a list.
#
# Theoretical number of comparisons for k smallest of n random inputs:
#
# Step Comparisons Action
# ---- -------------------------- ---------------------------
# 1 1.66 * k heapify the first k-inputs
# 2 n - k compare remaining elements to top of heap
# 3 k * (1 + lg2(k)) * ln(n/k) replace the topmost value on the heap
# 4 k * lg2(k) - (k/2) final sort of the k most extreme values
#
# Combining and simplifying for a rough estimate gives:
#
# comparisons = n + k * (log(k, 2) * log(n/k) + log(k, 2) + log(n/k))
I was looking at this pycon talk, 34:30 and the speaker says that getting the t largest elements of a list of n elements can be done in O(t + n).
How is that possible? My understanding is that creating the heap will be O(n), but what’s the complexity of nlargest itself, is it O(n + t) or O(t) (and what’s the actual algorithm)?
The speaker is wrong in this case. The actual cost is O(n * log(t)). Heapify is called only on the first t elements of the iterable. That’s O(t), but is insignificant if t is much smaller than n. Then all the remaining elements are added to this “little heap” via heappushpop, one at a time. That takes O(log(t)) time per invocation of heappushpop. The length of the heap remains t throughout. At the very end, the heap is sorted, which costs O(t * log(t)), but that’s also insignificant if t is much smaller than n.
Fun with Theory 😉
There are reasonably easy ways to find the t’th-largest element in expected O(n) time; for example, see here. There are harder ways to do it in worst-case O(n) time. Then, in another pass over the input, you could output the t elements >= the t-th largest (with tedious complications in case of duplicates). So the whole job can be done in O(n) time.
But those ways require O(n) memory too. Python doesn’t use them. An advantage of what’s actually implemented is that the worst-case “extra” memory burden is O(t), and that can be very significant when the input is, for example, a generator producing a great many values.
It’s actually O(n+tlog(n)) because heapify takes O(n) and for every element of largest or smallest takes O(log(n)). So for t largest/smallest it takes tlog(n). Therefore time complexity will be O(n+t*log(n))
For Heapq t largest or t smallest, the time complexity will be O(nlog(t))
Heapq will build the heap for the first t elements, then later on it will iterate over the remaining elements by pushing and popping the elements from the heap (maintaining the t elements in the heap).
- For building the heap for first t elements will be done
tlog(t) - For pushing and popping, the remaining elements will be done in
(n-t)log(t) - The overall time complexity will be
nlog(t)
According to the comments in the heapq source code:
# Algorithm notes for nlargest() and nsmallest()
# ==============================================
#
# Make a single pass over the data while keeping the k most extreme values
# in a heap. Memory consumption is limited to keeping k values in a list.
#
# Theoretical number of comparisons for k smallest of n random inputs:
#
# Step Comparisons Action
# ---- -------------------------- ---------------------------
# 1 1.66 * k heapify the first k-inputs
# 2 n - k compare remaining elements to top of heap
# 3 k * (1 + lg2(k)) * ln(n/k) replace the topmost value on the heap
# 4 k * lg2(k) - (k/2) final sort of the k most extreme values
#
# Combining and simplifying for a rough estimate gives:
#
# comparisons = n + k * (log(k, 2) * log(n/k) + log(k, 2) + log(n/k))