Count number of elements in each column less than x

Question

I have a DataFrame which looks like below. I am trying to count the number of elements less than 2.0 in each column, then I will visualize the result in a bar plot. I did it using lists and loops, but I wonder if there is a "Pandas way" to do this quickly.

x = []
for i in range(6):
    x.append(df[df.ix[:,i]<2.0].count()[i])

then I can get a bar plot using list x.

          A          B          C          D          E          F 
0       2.142      1.929      1.674      1.547      3.395      2.382
1       2.077      1.871      1.614      1.491      3.110      2.288
2       2.098      1.889      1.610      1.487      3.020      2.262
3       1.990      1.760      1.479      1.366      2.496      2.128
4       1.935      1.765      1.656      1.530      2.786      2.433

Asked By: marillion

||

Source

Answer 1

In [96]:

df = pd.DataFrame({'a':randn(10), 'b':randn(10), 'c':randn(10)})
df
Out[96]:
          a         b         c
0 -0.849903  0.944912  1.285790
1 -1.038706  1.445381  0.251002
2  0.683135 -0.539052 -0.622439
3 -1.224699 -0.358541  1.361618
4 -0.087021  0.041524  0.151286
5 -0.114031 -0.201018 -0.030050
6  0.001891  1.601687 -0.040442
7  0.024954 -1.839793  0.917328
8 -1.480281  0.079342 -0.405370
9  0.167295 -1.723555 -0.033937

[10 rows x 3 columns]
In [97]:

df[df > 1.0].count()

Out[97]:
a    0
b    2
c    2
dtype: int64

So in your case:

df[df < 2.0 ].count()

should work

EDIT

some timings

In [3]:

%timeit df[df < 1.0 ].count() 
%timeit (df < 1.0).sum()
%timeit (df < 1.0).apply(np.count_nonzero)
1000 loops, best of 3: 1.47 ms per loop
1000 loops, best of 3: 560 us per loop
1000 loops, best of 3: 529 us per loop

So @DSM’s suggestions are correct and much faster than my suggestion

Answered By: EdChum

Answer 2

Method-chaining is possible (comparison operators have their respective methods, e.g. < = lt(), <= = le()):

df.lt(2).sum()

If you have multiple conditions to consider, e.g. count the number of values between 2 and 10. Then you can use boolean operator on two boolean Serieses:

(df.gt(2) & df.lt(10)).sum()

or you can use pd.eval():

pd.eval("2 < df < 10").sum()

Count the number of values less than 2 or greater than 10:

(df.lt(2) | df.gt(10)).sum()
# or
pd.eval("df < 2 or df > 10").sum()

Answered By: cottontail

Count number of elements in each column less than x

Question:

Answers: