Converting year and day of year into datetime index in pandas
Question:
I have a dataframe:
year doy
2000 49
2000 65
2000 81
2001 97
2001 113
2001 129
2001 145
2001 161
I want to create a datetime index for this dataframe. Here is what I am doing:
df.index = pandas.DatetimeIndex(df['doy'].apply(lambda x: date(2000, 1, 1)+ relativedelta(days=int(x)-1)))
However, this creates a datetime index which only uses 2000 as year. How can I fix that?
Answers:
You can use NumPy datetime64/timedelta64 arithmetic to find the desired dates:
In [97]: (np.asarray(df['year'], dtype='datetime64[Y]')-1970)+(np.asarray(df['doy'], dtype='timedelta64[D]')-1)
Out[97]:
array(['2000-02-18', '2000-03-05', '2000-03-21', '2001-04-07',
'2001-04-23', '2001-05-09', '2001-05-25', '2001-06-10'], dtype='datetime64[D]')
Since composing dates given various parts of dates (e.g. years, months, days, weeks, hours, etc.) is a common problem, here is a utility function to make it easier:
def compose_date(years, months=1, days=1, weeks=None, hours=None, minutes=None,
seconds=None, milliseconds=None, microseconds=None, nanoseconds=None):
years = np.asarray(years) - 1970
months = np.asarray(months) - 1
days = np.asarray(days) - 1
types = ('<M8[Y]', '<m8[M]', '<m8[D]', '<m8[W]', '<m8[h]',
'<m8[m]', '<m8[s]', '<m8[ms]', '<m8[us]', '<m8[ns]')
vals = (years, months, days, weeks, hours, minutes, seconds,
milliseconds, microseconds, nanoseconds)
return sum(np.asarray(v, dtype=t) for t, v in zip(types, vals)
if v is not None)
df = pd.DataFrame({'doy': [49, 65, 81, 97, 113, 129, 145, 161],
'year': [2000, 2000, 2000, 2001, 2001, 2001, 2001, 2001]})
df.index = compose_date(df['year'], days=df['doy'])
yields
doy year
2000-02-18 49 2000
2000-03-05 65 2000
2000-03-21 81 2000
2001-04-07 97 2001
2001-04-23 113 2001
2001-05-09 129 2001
2001-05-25 145 2001
2001-06-10 161 2001
You can use the date specifier %j to extract the day of year. So combine the two columns, shift the year, and convert to datetime!
pd.to_datetime(df['year'] * 1000 + df['doy'], format='%Y%j')
returns
0 2000-02-18
1 2000-03-05
2 2000-03-21
3 2001-04-07
4 2001-04-23
5 2001-05-09
6 2001-05-25
7 2001-06-10
dtype: datetime64[ns]
With read_csv()
import pathlib
import pandas as pd
def yday2date(year, yday):
return pd.to_datetime(year + '-' + yday,
format='%Y-%j')
df = pd.read_csv('input.txt', sep='s+',
parse_dates={'date': ['year', 'doy']},
date_parser=yday2date,
index_col='date')
In pd.to_datetime() you may specify the format and origin date as:
pd.to_datetime(df['doy'], unit='D', origin=pd.Timestamp(df['year']))
This method does all the hard work for you.
Q: How to convert day of year to datetime?
A: Given arrays year and doy:
y = pd.to_datetime(year, format = '%Y')
td = pd.Series(doy).apply(lambda x: pd.Timedelta(x, unit = 'd'))
t = y + td
I have a dataframe:
year doy
2000 49
2000 65
2000 81
2001 97
2001 113
2001 129
2001 145
2001 161
I want to create a datetime index for this dataframe. Here is what I am doing:
df.index = pandas.DatetimeIndex(df['doy'].apply(lambda x: date(2000, 1, 1)+ relativedelta(days=int(x)-1)))
However, this creates a datetime index which only uses 2000 as year. How can I fix that?
You can use NumPy datetime64/timedelta64 arithmetic to find the desired dates:
In [97]: (np.asarray(df['year'], dtype='datetime64[Y]')-1970)+(np.asarray(df['doy'], dtype='timedelta64[D]')-1)
Out[97]:
array(['2000-02-18', '2000-03-05', '2000-03-21', '2001-04-07',
'2001-04-23', '2001-05-09', '2001-05-25', '2001-06-10'], dtype='datetime64[D]')
Since composing dates given various parts of dates (e.g. years, months, days, weeks, hours, etc.) is a common problem, here is a utility function to make it easier:
def compose_date(years, months=1, days=1, weeks=None, hours=None, minutes=None,
seconds=None, milliseconds=None, microseconds=None, nanoseconds=None):
years = np.asarray(years) - 1970
months = np.asarray(months) - 1
days = np.asarray(days) - 1
types = ('<M8[Y]', '<m8[M]', '<m8[D]', '<m8[W]', '<m8[h]',
'<m8[m]', '<m8[s]', '<m8[ms]', '<m8[us]', '<m8[ns]')
vals = (years, months, days, weeks, hours, minutes, seconds,
milliseconds, microseconds, nanoseconds)
return sum(np.asarray(v, dtype=t) for t, v in zip(types, vals)
if v is not None)
df = pd.DataFrame({'doy': [49, 65, 81, 97, 113, 129, 145, 161],
'year': [2000, 2000, 2000, 2001, 2001, 2001, 2001, 2001]})
df.index = compose_date(df['year'], days=df['doy'])
yields
doy year
2000-02-18 49 2000
2000-03-05 65 2000
2000-03-21 81 2000
2001-04-07 97 2001
2001-04-23 113 2001
2001-05-09 129 2001
2001-05-25 145 2001
2001-06-10 161 2001
You can use the date specifier %j to extract the day of year. So combine the two columns, shift the year, and convert to datetime!
pd.to_datetime(df['year'] * 1000 + df['doy'], format='%Y%j')
returns
0 2000-02-18
1 2000-03-05
2 2000-03-21
3 2001-04-07
4 2001-04-23
5 2001-05-09
6 2001-05-25
7 2001-06-10
dtype: datetime64[ns]
With read_csv()
import pathlib
import pandas as pd
def yday2date(year, yday):
return pd.to_datetime(year + '-' + yday,
format='%Y-%j')
df = pd.read_csv('input.txt', sep='s+',
parse_dates={'date': ['year', 'doy']},
date_parser=yday2date,
index_col='date')
In pd.to_datetime() you may specify the format and origin date as:
pd.to_datetime(df['doy'], unit='D', origin=pd.Timestamp(df['year']))
This method does all the hard work for you.
Q: How to convert day of year to datetime?
A: Given arrays year and doy:
y = pd.to_datetime(year, format = '%Y')
td = pd.Series(doy).apply(lambda x: pd.Timedelta(x, unit = 'd'))
t = y + td