Python: Elegant and efficient ways to mask a list
Question:
Example:
from __future__ import division
import numpy as np
n = 8
"""masking lists"""
lst = range(n)
print lst
# the mask (filter)
msk = [(el>3) and (el<=6) for el in lst]
print msk
# use of the mask
print [lst[i] for i in xrange(len(lst)) if msk[i]]
"""masking arrays"""
ary = np.arange(n)
print ary
# the mask (filter)
msk = (ary>3)&(ary<=6)
print msk
# use of the mask
print ary[msk] # very elegant
and the results are:
>>>
[0, 1, 2, 3, 4, 5, 6, 7]
[False, False, False, False, True, True, True, False]
[4, 5, 6]
[0 1 2 3 4 5 6 7]
[False False False False True True True False]
[4 5 6]
As you see the operation of masking on array is more elegant compared to list. If you try to use the array masking scheme on list you’ll get an error:
>>> lst[msk]
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
TypeError: only integer arrays with one element can be converted to an index
The question is to find an elegant masking for lists.
Updates:
The answer by jamylak was accepted for introducing compress however the points mentioned by Joel Cornett made the solution complete to a desired form of my interest.
>>> mlist = MaskableList
>>> mlist(lst)[msk]
>>> [4, 5, 6]
Answers:
If you are using numpy:
>>> import numpy as np
>>> a = np.arange(8)
>>> mask = np.array([False, False, False, False, True, True, True, False], dtype=np.bool)
>>> a[mask]
array([4, 5, 6])
If you are not using numpy you are looking for itertools.compress
>>> from itertools import compress
>>> a = range(8)
>>> mask = [False, False, False, False, True, True, True, False]
>>> list(compress(a, mask))
[4, 5, 6]
Since jamylak already answered the question with a practical answer, here is my example of a list with builtin masking support (totally unnecessary, btw):
from itertools import compress
class MaskableList(list):
def __getitem__(self, index):
try: return super(MaskableList, self).__getitem__(index)
except TypeError: return MaskableList(compress(self, index))
Usage:
>>> myList = MaskableList(range(10))
>>> myList
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> mask = [0, 1, 1, 0]
>>> myList[mask]
[1, 2]
Note that compress stops when either the data or the mask runs out. If you wish to keep the portion of the list that extends past the length of the mask, you could try something like:
from itertools import izip_longest
[i[0] for i in izip_longest(myList, mask[:len(myList)], fillvalue=True) if i[1]]
i don’t consider it elegant. It’s compact, but tends to be confusing, as the construct is very different than most languages.
As Rossum has said about language design, we spend more time reading it than writing it. The more obscure the construction of a line of code, the more confusing it becomes to others, who may lack familiarity with Python, even though they have full competency in any number of other languages.
Readability trumps short form notations everyday in the real world of servicing code. Just like fixing your car. Big drawings with lots of information make troubleshooting a lot easier.
For me, I would much rather troubleshoot someone’s code that uses the long form
print [lst[i] for i in xrange(len(lst)) if msk[i]]
than the numpy short notation mask. I don’t need to have any special knowledge of a specific Python package to interpret it.
If you are using Numpy, you can do it easily using Numpy array without installing any other library:
>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>> msk = [ True, False, False, True, True, True, True, False, False, False]
>> a = np.array(a) # convert list to numpy array
>> result = a[msk] # mask a
>> result.tolist()
[0, 3, 4, 5, 6]
You could also just use list and zip
- define a funcion
def masklist(mylist,mymask):
return [a for a,b in zip(mylist,mymask) if b]
- use it!
n = 8
lst = range(n)
msk = [(el>3) and (el<=6) for el in lst]
lst_msk = masklist(lst,msk)
print(lst_msk)
The following works perfectly well in Python 3:
np.array(lst)[msk]
If you need a list back as the result:
np.array(lst)[msk].tolist()
Example:
from __future__ import division
import numpy as np
n = 8
"""masking lists"""
lst = range(n)
print lst
# the mask (filter)
msk = [(el>3) and (el<=6) for el in lst]
print msk
# use of the mask
print [lst[i] for i in xrange(len(lst)) if msk[i]]
"""masking arrays"""
ary = np.arange(n)
print ary
# the mask (filter)
msk = (ary>3)&(ary<=6)
print msk
# use of the mask
print ary[msk] # very elegant
and the results are:
>>>
[0, 1, 2, 3, 4, 5, 6, 7]
[False, False, False, False, True, True, True, False]
[4, 5, 6]
[0 1 2 3 4 5 6 7]
[False False False False True True True False]
[4 5 6]
As you see the operation of masking on array is more elegant compared to list. If you try to use the array masking scheme on list you’ll get an error:
>>> lst[msk]
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
TypeError: only integer arrays with one element can be converted to an index
The question is to find an elegant masking for lists.
Updates:
The answer by jamylak was accepted for introducing compress however the points mentioned by Joel Cornett made the solution complete to a desired form of my interest.
>>> mlist = MaskableList
>>> mlist(lst)[msk]
>>> [4, 5, 6]
If you are using numpy:
>>> import numpy as np
>>> a = np.arange(8)
>>> mask = np.array([False, False, False, False, True, True, True, False], dtype=np.bool)
>>> a[mask]
array([4, 5, 6])
If you are not using numpy you are looking for itertools.compress
>>> from itertools import compress
>>> a = range(8)
>>> mask = [False, False, False, False, True, True, True, False]
>>> list(compress(a, mask))
[4, 5, 6]
Since jamylak already answered the question with a practical answer, here is my example of a list with builtin masking support (totally unnecessary, btw):
from itertools import compress
class MaskableList(list):
def __getitem__(self, index):
try: return super(MaskableList, self).__getitem__(index)
except TypeError: return MaskableList(compress(self, index))
Usage:
>>> myList = MaskableList(range(10))
>>> myList
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> mask = [0, 1, 1, 0]
>>> myList[mask]
[1, 2]
Note that compress stops when either the data or the mask runs out. If you wish to keep the portion of the list that extends past the length of the mask, you could try something like:
from itertools import izip_longest
[i[0] for i in izip_longest(myList, mask[:len(myList)], fillvalue=True) if i[1]]
i don’t consider it elegant. It’s compact, but tends to be confusing, as the construct is very different than most languages.
As Rossum has said about language design, we spend more time reading it than writing it. The more obscure the construction of a line of code, the more confusing it becomes to others, who may lack familiarity with Python, even though they have full competency in any number of other languages.
Readability trumps short form notations everyday in the real world of servicing code. Just like fixing your car. Big drawings with lots of information make troubleshooting a lot easier.
For me, I would much rather troubleshoot someone’s code that uses the long form
print [lst[i] for i in xrange(len(lst)) if msk[i]]
than the numpy short notation mask. I don’t need to have any special knowledge of a specific Python package to interpret it.
If you are using Numpy, you can do it easily using Numpy array without installing any other library:
>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>> msk = [ True, False, False, True, True, True, True, False, False, False]
>> a = np.array(a) # convert list to numpy array
>> result = a[msk] # mask a
>> result.tolist()
[0, 3, 4, 5, 6]
You could also just use list and zip
- define a funcion
def masklist(mylist,mymask):
return [a for a,b in zip(mylist,mymask) if b]
- use it!
n = 8
lst = range(n)
msk = [(el>3) and (el<=6) for el in lst]
lst_msk = masklist(lst,msk)
print(lst_msk)
The following works perfectly well in Python 3:
np.array(lst)[msk]
If you need a list back as the result:
np.array(lst)[msk].tolist()