Why does max() sometimes return nan and sometimes ignores it?
Question:
This question is motivated by an answer I gave a while ago.
Let’s say I have a dataframe like this
import numpy as np
import pandas as pd
df = pd.DataFrame({'a': [1, 2, np.nan], 'b': [3, np.nan, 10], 'c':[np.nan, 5, 34]})
a b c
0 1.0 3.0 NaN
1 2.0 NaN 5.0
2 NaN 10.0 34.0
and I want to replace the NaN by the maximum of the row, I can do
df.apply(lambda row: row.fillna(row.max()), axis=1)
which gives me the desired output
a b c
0 1.0 3.0 3.0
1 2.0 5.0 5.0
2 34.0 10.0 34.0
When I, however, use
df.apply(lambda row: row.fillna(max(row)), axis=1)
for some reason it is replaced correctly only in two of three cases:
a b c
0 1.0 3.0 3.0
1 2.0 5.0 5.0
2 NaN 10.0 34.0
Indeed, if I check by hand
max(df.iloc[0, :])
max(df.iloc[1, :])
max(df.iloc[2, :])
Then it prints
3.0
5.0
nan
When doing
df.iloc[0, :].max()
df.iloc[1, :].max()
df.iloc[2, :].max()
it prints the expected
3.0
5.0
34.0
My question is why max() fails in 1 of three cases but not in all 3. Why are the NaN sometimes ignored and sometimes not?
Answers:
In the first case you are using the numpy max function, which is aware of how to handle numpy.nan.
In the second case you are using the builtin max function from python. This is not aware of how to handle numpy.nan. Presumably this effect is due to the fact that any comparison (>, <, == etc.) of numpy.nan with a float leads to False. An obvious way to implement max would be to iterate the iterable (the row in this case) and check if each value is larger than the previous, and store it as the maximum value if so. Since this larger than comparison will always be False when one of the compared values is numpy.nan, whether the recorded maximum is the number you want or numpy.nan depends entirely on whether the first value is numpy.nan or not.
the two are different: max() vs df.max().
max(): python built-in function, it must be a non-empty iterable. Check here:
https://docs.python.org/2/library/functions.html#max
While pandas dataframe — df.max(skipna=..), there is a parameter called skipna, the default value is True, which means the NA/null values are excluded. Check here:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.max.html
The reason is that max works by taking the first value as the “max seen so far”, and then checking each other value to see if it is bigger than the max seen so far. But nan is defined so that comparisons with it always return False — that is, nan > 1 is false but 1 > nan is also false.
So if you start with nan as the first value in the array, every subsequent comparison will be check whether some_other_value > nan. This will always be false, so nan will retain its position as “max seen so far”. On the other hand, if nan is not the first value, then when it is reached, the comparison nan > max_so_far will again be false. But in this case that means the current “max seen so far” (which is not nan) will remain the max seen so far, so the nan will always be discarded.
This is due to the ordering of the elements in the list. First off, if you type
max([1, 2, np.nan])
The result is 2, while
max([np.nan, 2, 3])
gives np.nan. The reason for this is that the max function goes through the values in the list one by one with a comparison like this:
if a > b
now if we look at what we get when comparing to nan, both np.nan > 2 and 1 > np.nan both give False, so in one case the running maximum is replaced with nan and in the other it is not.
If possibly it’s inf issue, try to replace it as well as nan.
df[column] = df[column].replace([np.inf, -np.inf], 0.0)
df[column] = df[column].replace([np.nan, -np.nan], 0.0)
This question is motivated by an answer I gave a while ago.
Let’s say I have a dataframe like this
import numpy as np
import pandas as pd
df = pd.DataFrame({'a': [1, 2, np.nan], 'b': [3, np.nan, 10], 'c':[np.nan, 5, 34]})
a b c
0 1.0 3.0 NaN
1 2.0 NaN 5.0
2 NaN 10.0 34.0
and I want to replace the NaN by the maximum of the row, I can do
df.apply(lambda row: row.fillna(row.max()), axis=1)
which gives me the desired output
a b c
0 1.0 3.0 3.0
1 2.0 5.0 5.0
2 34.0 10.0 34.0
When I, however, use
df.apply(lambda row: row.fillna(max(row)), axis=1)
for some reason it is replaced correctly only in two of three cases:
a b c
0 1.0 3.0 3.0
1 2.0 5.0 5.0
2 NaN 10.0 34.0
Indeed, if I check by hand
max(df.iloc[0, :])
max(df.iloc[1, :])
max(df.iloc[2, :])
Then it prints
3.0
5.0
nan
When doing
df.iloc[0, :].max()
df.iloc[1, :].max()
df.iloc[2, :].max()
it prints the expected
3.0
5.0
34.0
My question is why max() fails in 1 of three cases but not in all 3. Why are the NaN sometimes ignored and sometimes not?
In the first case you are using the numpy max function, which is aware of how to handle numpy.nan.
In the second case you are using the builtin max function from python. This is not aware of how to handle numpy.nan. Presumably this effect is due to the fact that any comparison (>, <, == etc.) of numpy.nan with a float leads to False. An obvious way to implement max would be to iterate the iterable (the row in this case) and check if each value is larger than the previous, and store it as the maximum value if so. Since this larger than comparison will always be False when one of the compared values is numpy.nan, whether the recorded maximum is the number you want or numpy.nan depends entirely on whether the first value is numpy.nan or not.
the two are different: max() vs df.max().
max(): python built-in function, it must be a non-empty iterable. Check here:
https://docs.python.org/2/library/functions.html#max
While pandas dataframe — df.max(skipna=..), there is a parameter called skipna, the default value is True, which means the NA/null values are excluded. Check here:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.max.html
The reason is that max works by taking the first value as the “max seen so far”, and then checking each other value to see if it is bigger than the max seen so far. But nan is defined so that comparisons with it always return False — that is, nan > 1 is false but 1 > nan is also false.
So if you start with nan as the first value in the array, every subsequent comparison will be check whether some_other_value > nan. This will always be false, so nan will retain its position as “max seen so far”. On the other hand, if nan is not the first value, then when it is reached, the comparison nan > max_so_far will again be false. But in this case that means the current “max seen so far” (which is not nan) will remain the max seen so far, so the nan will always be discarded.
This is due to the ordering of the elements in the list. First off, if you type
max([1, 2, np.nan])
The result is 2, while
max([np.nan, 2, 3])
gives np.nan. The reason for this is that the max function goes through the values in the list one by one with a comparison like this:
if a > b
now if we look at what we get when comparing to nan, both np.nan > 2 and 1 > np.nan both give False, so in one case the running maximum is replaced with nan and in the other it is not.
If possibly it’s inf issue, try to replace it as well as nan.
df[column] = df[column].replace([np.inf, -np.inf], 0.0)
df[column] = df[column].replace([np.nan, -np.nan], 0.0)