Filter pandas dataframe on dates and wrong format

Question

I have a dataframe df with a date column of strings like this one :

Date
01/06/2022
03/07/2022
18/05/2022
12/02/2021
WK28
WK30
15/09/2021
09/02/2021
...

I want to update my dataframe with the last 6 months data AND the wrong format data (WK28, WK30…) like this :

Date
01/06/2022
03/07/2022
18/05/2022
WK28
WK30
...

I managed to keep the last 6 months dates by converting the column to Date format and computing a mask with a condition :

df['Dates']=pd.to_datetime(df['Dates'], errors='coerce', dayfirst=True)
mask = df['Dates'] >= pd.Timestamp((datetime.today() - timedelta(days=180)).date())
df = df[mask]

But how can I also keep the wrong format data ?

Asked By: Saguaro

||

Source

Answer 1

Use boolean indexing with 2 masks:

# save date as datetime in series
date = pd.to_datetime(df['Date'], errors='coerce', dayfirst=True)
# is it NaT?
m1 = date.isna()
# is it in the last 6 months?
m2 = date.ge(pd.to_datetime('today')-pd.DateOffset(months=6))

# if any condition is True, keep the row
out = df[m1|m2]

output:

         Date
0  01/06/2022
1  03/07/2022
2  18/05/2022
4        WK28
5        WK30

intermediate masks:

         Date     m1     m2  m1|m2
0  01/06/2022  False   True   True
1  03/07/2022  False   True   True
2  18/05/2022  False   True   True
3  12/02/2021  False  False  False
4        WK28   True  False   True
5        WK30   True  False   True
6  15/09/2021  False  False  False
7  09/02/2021  False  False  False

Answered By: mozway

Filter pandas dataframe on dates and wrong format

Question:

Answers: