Fastest way to zero out low values in array?
Question:
So, lets say I have 100,000 float arrays with 100 elements each. I need the highest X number of values, BUT only if they are greater than Y. Any element not matching this should be set to 0. What would be the fastest way to do this in Python? Order must be maintained. Most of the elements are already set to 0.
sample variables:
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
expected result:
array = [0, .25, 0, .15, .5, 0, 0, 0, 0, 0]
Answers:
The simplest way would be:
topX = sorted([x for x in array if x > lowValY], reverse=True)[highCountX-1]
print [x if x >= topX else 0 for x in array]
In pieces, this selects all the elements greater than lowValY:
[x for x in array if x > lowValY]
This array only contains the number of elements greater than the threshold. Then, sorting it so the largest values are at the start:
sorted(..., reverse=True)
Then a list index takes the threshold for the top highCountX elements:
sorted(...)[highCountX-1]
Finally, the original array is filled out using another list comprehension:
[x if x >= topX else 0 for x in array]
There is a boundary condition where there are two or more equal elements that (in your example) are 3rd highest elements. The resulting array will contain that element more than once.
There are other boundary conditions as well, such as if len(array) < highCountX. Handling such conditions is left to the implementor.
This is a typical job for NumPy, which is very fast for these kinds of operations:
array_np = numpy.asarray(array)
low_values_flags = array_np < lowValY # Where values are low
array_np[low_values_flags] = 0 # All low values set to 0
Now, if you only need the highCountX largest elements, you can even “forget” the small elements (instead of setting them to 0 and sorting them) and only sort the list of large elements:
array_np = numpy.asarray(array)
print numpy.sort(array_np[array_np >= lowValY])[-highCountX:]
Of course, sorting the whole array if you only need a few elements might not be optimal. Depending on your needs, you might want to consider the standard heapq module.
Settings elements below some threshold to zero is easy:
array = [ x if x > threshold else 0.0 for x in array ]
(plus the occasional abs() if needed.)
The requirement of the N highest numbers is a bit vague, however. What if there are e.g. N+1 equal numbers above the threshold? Which one to truncate?
You could sort the array first, then set the threshold to the value of the Nth element:
threshold = sorted(array, reverse=True)[N]
array = [ x if x >= threshold else 0.0 for x in array ]
Note: this solution is optimized for readability not performance.
Using numpy:
# assign zero to all elements less than or equal to `lowValY`
a[a<=lowValY] = 0
# find n-th largest element in the array (where n=highCountX)
x = partial_sort(a, highCountX, reverse=True)[:highCountX][-1]
#
a[a<x] = 0 #NOTE: it might leave more than highCountX non-zero elements
# . if there are duplicates
Where partial_sort could be:
def partial_sort(a, n, reverse=False):
#NOTE: in general it should return full list but in your case this will do
return sorted(a, reverse=reverse)[:n]
The expression a[a<value] = 0 can be written without numpy as follows:
for i, x in enumerate(a):
if x < value:
a[i] = 0
You can use map and lambda, it should be fast enough.
new_array = map(lambda x: x if x>y else 0, array)
Use a heap.
This works in time O(n*lg(HighCountX)).
import heapq
heap = []
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
for i in range(1,highCountX):
heappush(heap, lowValY)
heappop(heap)
for i in range( 0, len(array) - 1)
if array[i] > heap[0]:
heappush(heap, array[i])
min = heap[0]
array = [x if x >= min else 0 for x in array]
deletemin works in heap O(lg(k)) and insertion O(lg(k)) or O(1) depending on which heap type you use.
There’s a special MaskedArray class in NumPy that does exactly that. You can “mask” elements based on any precondition. This better represent your need than assigning zeroes: numpy operations will ignore masked values when appropriate (for example, finding mean value).
>>> from numpy import ma
>>> x = ma.array([.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0])
>>> x1 = ma.masked_inside(0, 0.1) # mask everything in 0..0.1 range
>>> x1
masked_array(data = [-- 0.25 -- 0.15 0.5 -- -- -- -- --],
mask = [ True False True False False True True True True True],
fill_value = 1e+20)
>>> print x.filled(0) # Fill with zeroes
[ 0 0.25 0 0.15 0.5 0 0 0 0 0 ]
As an addded benefit, masked arrays are well supported in matplotlib visualisation library if you need this.
Using a heap is a good idea, as egon says. But you can use the heapq.nlargest function to cut down on some effort:
import heapq
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
threshold = max(heapq.nlargest(highCountX, array)[-1], lowValY)
array = [x if x >= threshold else 0 for x in array]
from scipy.stats import threshold
thresholded = threshold(array, 0.5)
🙂
So, lets say I have 100,000 float arrays with 100 elements each. I need the highest X number of values, BUT only if they are greater than Y. Any element not matching this should be set to 0. What would be the fastest way to do this in Python? Order must be maintained. Most of the elements are already set to 0.
sample variables:
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
expected result:
array = [0, .25, 0, .15, .5, 0, 0, 0, 0, 0]
The simplest way would be:
topX = sorted([x for x in array if x > lowValY], reverse=True)[highCountX-1]
print [x if x >= topX else 0 for x in array]
In pieces, this selects all the elements greater than lowValY:
[x for x in array if x > lowValY]
This array only contains the number of elements greater than the threshold. Then, sorting it so the largest values are at the start:
sorted(..., reverse=True)
Then a list index takes the threshold for the top highCountX elements:
sorted(...)[highCountX-1]
Finally, the original array is filled out using another list comprehension:
[x if x >= topX else 0 for x in array]
There is a boundary condition where there are two or more equal elements that (in your example) are 3rd highest elements. The resulting array will contain that element more than once.
There are other boundary conditions as well, such as if len(array) < highCountX. Handling such conditions is left to the implementor.
This is a typical job for NumPy, which is very fast for these kinds of operations:
array_np = numpy.asarray(array)
low_values_flags = array_np < lowValY # Where values are low
array_np[low_values_flags] = 0 # All low values set to 0
Now, if you only need the highCountX largest elements, you can even “forget” the small elements (instead of setting them to 0 and sorting them) and only sort the list of large elements:
array_np = numpy.asarray(array)
print numpy.sort(array_np[array_np >= lowValY])[-highCountX:]
Of course, sorting the whole array if you only need a few elements might not be optimal. Depending on your needs, you might want to consider the standard heapq module.
Settings elements below some threshold to zero is easy:
array = [ x if x > threshold else 0.0 for x in array ]
(plus the occasional abs() if needed.)
The requirement of the N highest numbers is a bit vague, however. What if there are e.g. N+1 equal numbers above the threshold? Which one to truncate?
You could sort the array first, then set the threshold to the value of the Nth element:
threshold = sorted(array, reverse=True)[N]
array = [ x if x >= threshold else 0.0 for x in array ]
Note: this solution is optimized for readability not performance.
Using numpy:
# assign zero to all elements less than or equal to `lowValY`
a[a<=lowValY] = 0
# find n-th largest element in the array (where n=highCountX)
x = partial_sort(a, highCountX, reverse=True)[:highCountX][-1]
#
a[a<x] = 0 #NOTE: it might leave more than highCountX non-zero elements
# . if there are duplicates
Where partial_sort could be:
def partial_sort(a, n, reverse=False):
#NOTE: in general it should return full list but in your case this will do
return sorted(a, reverse=reverse)[:n]
The expression a[a<value] = 0 can be written without numpy as follows:
for i, x in enumerate(a):
if x < value:
a[i] = 0
You can use map and lambda, it should be fast enough.
new_array = map(lambda x: x if x>y else 0, array)
Use a heap.
This works in time O(n*lg(HighCountX)).
import heapq
heap = []
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
for i in range(1,highCountX):
heappush(heap, lowValY)
heappop(heap)
for i in range( 0, len(array) - 1)
if array[i] > heap[0]:
heappush(heap, array[i])
min = heap[0]
array = [x if x >= min else 0 for x in array]
deletemin works in heap O(lg(k)) and insertion O(lg(k)) or O(1) depending on which heap type you use.
There’s a special MaskedArray class in NumPy that does exactly that. You can “mask” elements based on any precondition. This better represent your need than assigning zeroes: numpy operations will ignore masked values when appropriate (for example, finding mean value).
>>> from numpy import ma
>>> x = ma.array([.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0])
>>> x1 = ma.masked_inside(0, 0.1) # mask everything in 0..0.1 range
>>> x1
masked_array(data = [-- 0.25 -- 0.15 0.5 -- -- -- -- --],
mask = [ True False True False False True True True True True],
fill_value = 1e+20)
>>> print x.filled(0) # Fill with zeroes
[ 0 0.25 0 0.15 0.5 0 0 0 0 0 ]
As an addded benefit, masked arrays are well supported in matplotlib visualisation library if you need this.
Using a heap is a good idea, as egon says. But you can use the heapq.nlargest function to cut down on some effort:
import heapq
array = [.06, .25, 0, .15, .5, 0, 0, 0.04, 0, 0]
highCountX = 3
lowValY = .1
threshold = max(heapq.nlargest(highCountX, array)[-1], lowValY)
array = [x if x >= threshold else 0 for x in array]
from scipy.stats import threshold
thresholded = threshold(array, 0.5)
🙂