find the set of column indices for non-zero values in each row in pandas' data frame

Question

Is there a good way to find the set of column indices for non-zero values in each row in pandas’ data frame? Do I have to traverse the data frame row-by-row?

For example, the data frame is

c1  c2  c3  c4 c5 c6 c7 c8  c9
 1   1   0   0  0  0  0  0   0
 1   0   0   0  0  0  0  0   0
 0   1   0   0  0  0  0  0   0
 1   0   0   0  0  0  0  0   0
 0   1   0   0  0  0  0  0   0
 0   0   0   0  0  0  0  0   0
 0   2   1   1  1  1  1  0   2
 1   5   5   0  0  1  0  4   6
 4   3   0   1  1  1  1  5  10
 3   5   2   4  1  2  2  1   3
 6   4   0   1  0  0  0  0   0
 3   9   1   0  1  0  2  1   0

The output is expected to be

['c1','c2']
['c1']
['c2']
...

Asked By: Qiang Li

||

Source

Answer 1

Potentially a better data structure (rather than a Series of lists) is to stack:

In [11]: res = df[df!=0].stack()

In [12]: res
Out[12]:
0   c1     1
    c2     1
1   c1     1
2   c2     1
3   c1     1
...

And you can iterate over the original rows:

In [13]: res.loc[0]
Out[13]:
c1    1
c2    1
dtype: float64

In [14]: res.loc[0].index
Out[14]: Index(['c1', 'c2'], dtype='object')

Note: I thought you used to be able to return a list in an apply (to create a DataFrame which has list elements) this no longer appears to be the case.

Answered By: Andy Hayden

Answer 2

It seems you have to traverse the DataFrame by row.

cols = df.columns
bt = df.apply(lambda x: x > 0)
bt.apply(lambda x: list(cols[x.values]), axis=1)

and you will get:

0                                 [c1, c2]
1                                     [c1]
2                                     [c2]
3                                     [c1]
4                                     [c2]
5                                       []
6             [c2, c3, c4, c5, c6, c7, c9]
7                 [c1, c2, c3, c6, c8, c9]
8         [c1, c2, c4, c5, c6, c7, c8, c9]
9     [c1, c2, c3, c4, c5, c6, c7, c8, c9]
10                            [c1, c2, c4]
11                [c1, c2, c3, c5, c7, c8]
dtype: object

If performance is matter, try to pass raw=True to boolean DataFrame creation like below:

%timeit df.apply(lambda x: x > 0, raw=True).apply(lambda x: list(cols[x.values]), axis=1)
1000 loops, best of 3: 812 µs per loop

It brings you a better performance gain. Following is raw=False (which is default) result:

%timeit df.apply(lambda x: x > 0).apply(lambda x: list(cols[x.values]), axis=1)
100 loops, best of 3: 2.59 ms per loop

Answered By: Younggun Kim

Answer 3

How about this approach?

#create a True / False data frame
df_boolean = df>0

#a little helper method that uses boolean slicing internally 
def bar(x,columns):
    return ','.join(list(columns[x]))

#use an apply along the column axis
df_boolean['result'] = df_boolean.apply(lambda x: bar(x,df_boolean.columns),axis=1)

# filter out the empty "rows" adn grab the result column
df_result =  df_boolean[df_boolean['result'] != '']['result']

#append an axis, just so each line will will output a list 
lst_result = df_result.values[:,np.newaxis]

print 'n'.join([ str(myelement) for myelement in lst_result])

and this produces:

['c1,c2']
['c1']
['c2']
['c1']
['c2']
['c2,c3,c4,c5,c6,c7,c9']
['c1,c2,c3,c6,c8,c9']
['c1,c2,c4,c5,c6,c7,c8,c9']
['c1,c2,c3,c4,c5,c6,c7,c8,c9']
['c1,c2,c4']
['c1,c2,c3,c5,c7,c8']

Answered By: Dickster

Answer 4

If you only want to locate the nonzero values, both numpy.argwhere() and nonzero() are one-liners.

nzero = np.argwhere(df.to_numpy())
# nzero is an array of two-element arrays [irow, icol]
nz = df.to_numpy().nonzero()
# Alternatively, nz is a duple of numpy 1D-arrays of corresponding indices

But to get the row-wise output requested, I can’t think of a way to avoid a loop over rows. The accepted answer is much shorter.

pairit = iter(nzero)
pair = next(pairit)
for irow in range(len(df)):
    # want one list for each row
    cols = []
    while pair[0] == irow:
        cols.append(df.columns[pair[1]])
        try:
            pair = next(pairit)
        except StopIteration:
            break
    print(irow, cols)

Answered By: Karl I.

find the set of column indices for non-zero values in each row in pandas' data frame

Question:

Answers: