Why does pandas apply calculate twice

Question

I’m using the apply method on a panda’s DataFrame object. When my DataFrame has a single column, it appears that the applied function is being called twice. The questions are why? And, can I stop that behavior?

Code:

import pandas as pd

def mul2(x):
    print ('hello')
    return 2*x

df = pd.DataFrame({'a': [1,2,0.67,1.34]})
df.apply(mul2)

Output:

hello
hello

0  2.00
1  4.00
2  1.34
3  2.68

I’m printing ‘hello’ from within the function being applied. I know it’s being applied twice because ‘hello’ printed twice. What’s more is that if I had two columns, ‘hello’ prints 3 times. Even more still is when I call applied to just the column ‘hello’ prints 4 times.

Code:

df.a.apply(mul2)

Output:

hello
hello
hello
hello
0    2.00
1    4.00
2    1.34
3    2.68
Name: a, dtype: float64

Asked By: piRSquared

||

Source

Answer 1

Probably related to this issue. With groupby, the applied function is called one extra time to see if certain optimizations can be done. I’d guess something similar is going on here. It doesn’t look like there’s any way around it at the moment (although I could be wrong about the source of the behavior you’re seeing). Is there a reason you need it to not do that extra call.

Also, calling it four times when you apply on the column is normal. When you get one columnm you get a Series, not a DataFrame. apply on a Series applies the function to each element. Since your column has four elements in it, the function is called four times.

Answered By: BrenBarn

Answer 2

This behavior is intended, as an optimization.

See the docs:

In the current implementation apply calls func twice on the first
column/row to decide whether it can take a fast or slow code path.
This can lead to unexpected behavior if func has side-effects, as they
will take effect twice for the first column/row.

Answered By: MERose

Answer 3

This behavior has been fixed with pandas 1.1, please upgrade!

Now, apply and applymap on DataFrame evaluates first row/column only once.

Initially, we had GroupBy.apply and Series/df.apply evaluating the first group twice. The reason the first group is evaluated twice is because apply wants to know whether it can "optimize" the calculation (sometimes this is possible if apply receives a numpy or cythonized function). With pandas 0.25, this behavior was fixed for GroupBy.apply. Now, with pandas 1.1, this will also be fixed for df.apply.

Old Behavior [pandas <= 1.0.X]

pd.__version__ 
# '1.0.4'

df.apply(mul2)
hello
hello

      a
0  2.00
1  4.00
2  1.34
3  2.68

New Behavior [pandas >= 1.1]

pd.__version__
# '1.1.0.dev0+2004.g8d10bfb6f'

df.apply(mul2)
hello

      a
0  2.00
1  4.00
2  1.34
3  2.68

Answered By: cs95

Why does pandas apply calculate twice

Question:

Answers:

This behavior has been fixed with pandas 1.1, please upgrade!