Set the format for pd.to_datetime
Question:
Hi already referred to this post but I could not get through my issue. I have a column in my CSV which is string and the sample values are below (note that the month
and year
positioning are sometimes reversed). What format do I need to set in my to_datetime
? I tried all the below approaches
df = pd.read_csv("filename.csv") #Imagine there is a Month column
#[1] df["Month"] = pd.to_datetime(df["Month"])
#[2] df["Month"] = pd.to_datetime(df["Month"], format="%m/%d/%Y")
[Month]
Mar-97
Apr-97
May-97
Jun-97
Nov-00
Dec-00
1-Jan
1-Feb
1-Mar
1-Apr
I get the error
ValueError: day is out of range for month
for [1] and I get
ValueError: time data ‘Mar-97’ does not match format ‘%m/%d/%Y’ (match)
for [2]. I tried to remove the %d
too but no luck. Could you please point me what is going wrong here.
Answers:
One way is to use try
/ except
with pd.Series.apply
:
s = pd.Series(['Mar-97', 'May-97', 'Nov-00', '1-Jan', '1-Mar'])
def converter(x):
try:
return pd.datetime.strptime(x, '%b-%y')
except ValueError:
year, month = x.split('-') # split by delimiter
x = year.zfill(2) + '-' + month # %y requires 0-padding
return pd.datetime.strptime(x, '%y-%b')
res = s.apply(converter)
print(res)
0 1997-03-01
1 1997-05-01
2 2000-11-01
3 2001-01-01
4 2001-03-01
dtype: datetime64[ns]
Since we have defined converter
as a function, we can use this directly with pd.read_csv
:
df = pd.read_csv('file.csv', parse_dates=['dt_col_name'], date_parser=converter)
Python’s strftime directives is a useful reference for constructing datetime
format strings.
Not the most elegant, but you might try to fix and order the year and month parts. The below code works:
Recreate your data:
df = pd.DataFrame({"date_str": ['Mar-97', 'Apr-97', 'May-97',
'Jun-97', 'Nov-00', 'Dec-00',
'1-Jan', '1-Feb', '1-Mar', '1-Apr']})
Split parts:
df = pd.concat([df, df['date_str'].str.split("-", expand=True)], axis=1)
Organize month and year:
df.loc[df[0].str.len() == 3, 'month'] = df.loc[df[0].str.len() == 3, 0]
df.loc[df[1].str.len() == 3, 'month'] = df.loc[df[1].str.len() == 3, 1]
df.loc[df[0].str.len() != 3, 'year'] = df.loc[df[0].str.len() != 3, 0]
df.loc[df[1].str.len() != 3, 'year'] = df.loc[df[1].str.len() != 3, 1]
Correct years that are only a single digit:
df.loc[df['year'].str.len() == 1, 'year'] = '0' + df.loc[df['year'].str.len() == 1, 'year']
Generate proper date column:
df['date'] = (df['month'] + '-' + df['year']).apply(lambda x: pd.to_datetime(x, format="%b-%y"))
Output:
print(df[‘date’])
0 1997-03-01
1 1997-04-01
2 1997-05-01
3 1997-06-01
4 2000-11-01
5 2000-12-01
6 2001-01-01
7 2001-02-01
8 2001-03-01
9 2001-04-01
Name: date, dtype: datetime64[ns]
Any of this worked for me while using pandas. I fix this problem and other pretty similars of dtypes with this function:
dfReCajatot2["Column dates"] = dfReCajatot2["Column dates""].apply(pd.to_datetime, errors='coerce')
This changes the tipe of each cell instead of all column. Maybe some values throws diferent structure but you can fix that later
Hi already referred to this post but I could not get through my issue. I have a column in my CSV which is string and the sample values are below (note that the month
and year
positioning are sometimes reversed). What format do I need to set in my to_datetime
? I tried all the below approaches
df = pd.read_csv("filename.csv") #Imagine there is a Month column
#[1] df["Month"] = pd.to_datetime(df["Month"])
#[2] df["Month"] = pd.to_datetime(df["Month"], format="%m/%d/%Y")
[Month]
Mar-97
Apr-97
May-97
Jun-97
Nov-00
Dec-00
1-Jan
1-Feb
1-Mar
1-Apr
I get the error
ValueError: day is out of range for month
for [1] and I get
ValueError: time data ‘Mar-97’ does not match format ‘%m/%d/%Y’ (match)
for [2]. I tried to remove the %d
too but no luck. Could you please point me what is going wrong here.
One way is to use try
/ except
with pd.Series.apply
:
s = pd.Series(['Mar-97', 'May-97', 'Nov-00', '1-Jan', '1-Mar'])
def converter(x):
try:
return pd.datetime.strptime(x, '%b-%y')
except ValueError:
year, month = x.split('-') # split by delimiter
x = year.zfill(2) + '-' + month # %y requires 0-padding
return pd.datetime.strptime(x, '%y-%b')
res = s.apply(converter)
print(res)
0 1997-03-01
1 1997-05-01
2 2000-11-01
3 2001-01-01
4 2001-03-01
dtype: datetime64[ns]
Since we have defined converter
as a function, we can use this directly with pd.read_csv
:
df = pd.read_csv('file.csv', parse_dates=['dt_col_name'], date_parser=converter)
Python’s strftime directives is a useful reference for constructing datetime
format strings.
Not the most elegant, but you might try to fix and order the year and month parts. The below code works:
Recreate your data:
df = pd.DataFrame({"date_str": ['Mar-97', 'Apr-97', 'May-97',
'Jun-97', 'Nov-00', 'Dec-00',
'1-Jan', '1-Feb', '1-Mar', '1-Apr']})
Split parts:
df = pd.concat([df, df['date_str'].str.split("-", expand=True)], axis=1)
Organize month and year:
df.loc[df[0].str.len() == 3, 'month'] = df.loc[df[0].str.len() == 3, 0]
df.loc[df[1].str.len() == 3, 'month'] = df.loc[df[1].str.len() == 3, 1]
df.loc[df[0].str.len() != 3, 'year'] = df.loc[df[0].str.len() != 3, 0]
df.loc[df[1].str.len() != 3, 'year'] = df.loc[df[1].str.len() != 3, 1]
Correct years that are only a single digit:
df.loc[df['year'].str.len() == 1, 'year'] = '0' + df.loc[df['year'].str.len() == 1, 'year']
Generate proper date column:
df['date'] = (df['month'] + '-' + df['year']).apply(lambda x: pd.to_datetime(x, format="%b-%y"))
Output:
print(df[‘date’])
0 1997-03-01
1 1997-04-01
2 1997-05-01
3 1997-06-01
4 2000-11-01
5 2000-12-01
6 2001-01-01
7 2001-02-01
8 2001-03-01
9 2001-04-01
Name: date, dtype: datetime64[ns]
Any of this worked for me while using pandas. I fix this problem and other pretty similars of dtypes with this function:
dfReCajatot2["Column dates"] = dfReCajatot2["Column dates""].apply(pd.to_datetime, errors='coerce')
This changes the tipe of each cell instead of all column. Maybe some values throws diferent structure but you can fix that later