Dataframe : replace value and values around based on condition

Question

I would like to create a filter to replace values in a dataframe column based on a condition and also the values around it.

For exemple I would like to filter values and replace then with NaN if they are superior to 45 but also the value before and after it even if they are not meeting the condition:

df[i] = 10, 12, 25, 60, 32, 26, 23

In this exemple the filter should replace 60 by NaN and also the value before (25) and the value after (32).The result of the filter would be :

df[i] = 10, 12, NaN, NaN, NaN, 26, 23

So far I am using this line but it only replace value that meet the condition and not also values around:

df[i].where(df[i] <= 45, np.nan, inplace=True)

Asked By: Ketchup

||

Source

Answer 1

You can compare original with shifted values of mask chained by | for bitwise OR:

m = df['i'].gt(45)

mask = m.shift(fill_value=False) | m.shift(-1, fill_value=False) | m

#alternative solution +1, -1 value by parameter limit
#mask = df['i'].where(m).ffill(limit=1).bfill(limit=1).notna()

df.loc[mask, 'i'] = np.nan

Another idea for general mask (but slowier like solution above):

mask = (df['i'].rolling(3, min_periods=1, center=True)
               .apply(lambda x: (x>45).any()).astype(bool))

df.loc[mask, 'i'] = np.nan

Answered By: jezrael

Answer 2

You can use boolean mask to filter the dataframe column and replace the value with Nan.

mask = (df["col"] > 45)
df.loc[mask, "col"] = np.nan

After this you can use the mask to replace the values immediately before and after each value that met the condition with NaN as well.

df.loc[(mask.shift(1) | mask) | (mask.shift(-1)), "col"] = np.nan

Answered By: Data Guy

Dataframe : replace value and values around based on condition

Question:

Answers: