Numpy Indexing: Return the rest
Question:
A simply example of numpy indexing:
In: a = numpy.arange(10)
In: sel_id = numpy.arange(5)
In: a[sel_id]
Out: array([0,1,2,3,4])
How do I return the rest of the array that are not indexed by sel_id? What I can think of is:
In: numpy.array([x for x in a if x not in a[id]])
out: array([5,6,7,8,9])
Is there any easier way?
Answers:
numpy.setdiff1d(a, a[sel_id]) should do the trick. Don’t know if there’s something neater than this.
It’s more like:
a = numpy.array([1, 2, 3, 4, 5, 6, 7, 4])
exclude_index = numpy.arange(5)
include_index = numpy.setdiff1d(numpy.arange(len(a)), exclude_index)
a[include_index]
# array([6, 7, 4])
# Notice this is a little different from
numpy.setdiff1d(a, a[exclude_index])
# array([6, 7]
Also, if they are contiguous use the [N:] syntax to select the rest. For instance, arr[5:] would select the 5th to last element in the array.
For this simple 1D case, I’d actually use a boolean mask:
a = numpy.arange(10)
include_index = numpy.arange(4)
include_idx = set(include_index) #Set is more efficient, but doesn't reorder your elements if that is desireable
mask = numpy.array([(i in include_idx) for i in xrange(len(a))])
Now you can get your values:
included = a[mask] # array([0, 1, 2, 3])
excluded = a[~mask] # array([4, 5, 6, 7, 8, 9])
Note that a[mask] doesn’t necessarily yield the same thing as a[include_index] since the order of include_index matters for the output in that scenario (it should be roughly equivalent to a[sorted(include_index)]). However, since the order of your excluded items isn’t well defined, this should work Ok.
EDIT
A better way to create the mask is:
mask = np.zeros(a.shape,dtype=bool)
mask[include_idx] = True
(thanks to seberg).
You can do this nicely with boolean masks:
a = numpy.arange(10)
mask = np.ones(len(a), dtype=bool) # all elements included/True.
mask[[7,2,8]] = False # Set unwanted elements to False
print a[mask]
# Gives (removing entries 7, 2 and 8):
[0 1 3 4 5 6 9]
Addition (taken from @mgilson). The binary mask created can be used nicely to get back the original slices with a[~mask] however this is only the same if the original indices were sorted.
EDIT: Moved down, as I had to realize that I would consider np.delete buggy at this time (Sep. 2012).
You could also use np.delete, though masks are more powerful (and in the future I think that should be an OK option). At the moment however its slower then the above, and will create unexpected results with negative indices (or steps when given a slice).
print np.delete(a, [7,2,8])
Assuming that a is a 1D array, you could just pop the items you don’t want from the list of indices:
accept = [i for i in range(a.size) if i not in avoid_list]
a[accept]
You could also try to use something like
accept = sorted(set(range(a.size)) - set(indices_to_discard))
a[accept]
The idea is to use fancy indexing on the complementary of the set of indices you don’t want.
I would do this with a Boolean mask but a little different. Which has the benefit of working in N-dimensions, with continuous or not indices. Memory usage will depend on if a view or copy is made for the masked array and I am not sure.
import numpy
a = numpy.arange(10)
sel_id = numpy.arange(5)
mask = numpy.ma.make_mask_none(a.shape)
mask[sel_id] = True
answer = numpy.ma.masked_array(a, mask).compressed()
print answer
# [5 6 7 8 9]
Here’s another way, using numpy’s isin() function:
import numpy as np
a = np.arange(10)
sel_id = np.arange(5)
a[~np.isin(np.arange(a.size), sel_id)]
Explanation:
np.arange(a.size) gives all the indices of a, i.e. [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
np.isin(np.arange(a.size), sel_id) returns a boolean mask [ True, True, True, True, True, False, False, False, False, False] with True at indices which are in sel_id and False otherwise. Since we want to get the indices that are not in sel_id we use the bitwise NOT operator ~ to invert the boolean mask.
A simply example of numpy indexing:
In: a = numpy.arange(10)
In: sel_id = numpy.arange(5)
In: a[sel_id]
Out: array([0,1,2,3,4])
How do I return the rest of the array that are not indexed by sel_id? What I can think of is:
In: numpy.array([x for x in a if x not in a[id]])
out: array([5,6,7,8,9])
Is there any easier way?
numpy.setdiff1d(a, a[sel_id]) should do the trick. Don’t know if there’s something neater than this.
It’s more like:
a = numpy.array([1, 2, 3, 4, 5, 6, 7, 4])
exclude_index = numpy.arange(5)
include_index = numpy.setdiff1d(numpy.arange(len(a)), exclude_index)
a[include_index]
# array([6, 7, 4])
# Notice this is a little different from
numpy.setdiff1d(a, a[exclude_index])
# array([6, 7]
Also, if they are contiguous use the [N:] syntax to select the rest. For instance, arr[5:] would select the 5th to last element in the array.
For this simple 1D case, I’d actually use a boolean mask:
a = numpy.arange(10)
include_index = numpy.arange(4)
include_idx = set(include_index) #Set is more efficient, but doesn't reorder your elements if that is desireable
mask = numpy.array([(i in include_idx) for i in xrange(len(a))])
Now you can get your values:
included = a[mask] # array([0, 1, 2, 3])
excluded = a[~mask] # array([4, 5, 6, 7, 8, 9])
Note that a[mask] doesn’t necessarily yield the same thing as a[include_index] since the order of include_index matters for the output in that scenario (it should be roughly equivalent to a[sorted(include_index)]). However, since the order of your excluded items isn’t well defined, this should work Ok.
EDIT
A better way to create the mask is:
mask = np.zeros(a.shape,dtype=bool)
mask[include_idx] = True
(thanks to seberg).
You can do this nicely with boolean masks:
a = numpy.arange(10)
mask = np.ones(len(a), dtype=bool) # all elements included/True.
mask[[7,2,8]] = False # Set unwanted elements to False
print a[mask]
# Gives (removing entries 7, 2 and 8):
[0 1 3 4 5 6 9]
Addition (taken from @mgilson). The binary mask created can be used nicely to get back the original slices with a[~mask] however this is only the same if the original indices were sorted.
EDIT: Moved down, as I had to realize that I would consider np.delete buggy at this time (Sep. 2012).
You could also use np.delete, though masks are more powerful (and in the future I think that should be an OK option). At the moment however its slower then the above, and will create unexpected results with negative indices (or steps when given a slice).
print np.delete(a, [7,2,8])
Assuming that a is a 1D array, you could just pop the items you don’t want from the list of indices:
accept = [i for i in range(a.size) if i not in avoid_list]
a[accept]
You could also try to use something like
accept = sorted(set(range(a.size)) - set(indices_to_discard))
a[accept]
The idea is to use fancy indexing on the complementary of the set of indices you don’t want.
I would do this with a Boolean mask but a little different. Which has the benefit of working in N-dimensions, with continuous or not indices. Memory usage will depend on if a view or copy is made for the masked array and I am not sure.
import numpy
a = numpy.arange(10)
sel_id = numpy.arange(5)
mask = numpy.ma.make_mask_none(a.shape)
mask[sel_id] = True
answer = numpy.ma.masked_array(a, mask).compressed()
print answer
# [5 6 7 8 9]
Here’s another way, using numpy’s isin() function:
import numpy as np
a = np.arange(10)
sel_id = np.arange(5)
a[~np.isin(np.arange(a.size), sel_id)]
Explanation:
np.arange(a.size) gives all the indices of a, i.e. [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
np.isin(np.arange(a.size), sel_id) returns a boolean mask [ True, True, True, True, True, False, False, False, False, False] with True at indices which are in sel_id and False otherwise. Since we want to get the indices that are not in sel_id we use the bitwise NOT operator ~ to invert the boolean mask.