Pandas: Convert Timestamp to datetime.date
Question:
I have a pandas column of Timestamp data
In [27]: train["Original_Quote_Date"][6]
Out[27]: Timestamp('2013-12-25 00:00:00')
How can check equivalence of these objects to datetime.date objects of the type
datetime.date(2013, 12, 25)
Answers:
You can convert a datetime.date object into a pandas Timestamp like this:
#!/usr/bin/env python3
# coding: utf-8
import pandas as pd
import datetime
# create a datetime data object
d_time = datetime.date(2010, 11, 12)
# create a pandas Timestamp object
t_stamp = pd.to_datetime('2010/11/12')
# cast `datetime_timestamp` as Timestamp object and compare
d_time2t_stamp = pd.to_datetime(d_time)
# print to double check
print(d_time)
print(t_stamp)
print(d_time2t_stamp)
# since the conversion succeds this prints `True`
print(d_time2t_stamp == t_stamp)
Use the .date method:
In [11]: t = pd.Timestamp('2013-12-25 00:00:00')
In [12]: t.date()
Out[12]: datetime.date(2013, 12, 25)
In [13]: t.date() == datetime.date(2013, 12, 25)
Out[13]: True
To compare against a DatetimeIndex (i.e. an array of Timestamps), you’ll want to do it the other way around:
In [21]: pd.Timestamp(datetime.date(2013, 12, 25))
Out[21]: Timestamp('2013-12-25 00:00:00')
In [22]: ts = pd.DatetimeIndex([t])
In [23]: ts == pd.Timestamp(datetime.date(2013, 12, 25))
Out[23]: array([ True], dtype=bool)
As of pandas 0.20.3, use .to_pydatetime() to convert any pandas.DateTimeIndex instances to Python datetime.datetime.
Assume time column is in timestamp integer msec format
1 day = 86400000 ms
Here you go:
day_divider = 86400000
df['time'] = df['time'].values.astype(dtype='datetime64[ms]') # for msec format
df['time'] = (df['time']/day_divider).values.astype(dtype='datetime64[D]') # for day format
So, got this from an IBM coursera tutorial.
data['date'] = data['TimeStamp'].apply(lambda d: datetime.date.fromtimestamp(d))
I’ve used the way recommended by Filomeno Gonzalez, albeit with a slight twist:
data['date'] = data['date'].apply(lambda x: x.date())
If I have a pandas DataFrame with timestamp column (1546300800000, 1546301100000, 1546301400000, 1546301700000, 1546302000000) and I want to convert this into date time format
import datetime
df['date'] = df['date'].apply(lambda x: datetime.datetime.fromtimestamp(x/1000.0))
This will return a column with the format 2019-01-01 00:00:00, 2019-01-01 00:05:00, 2019-01-01 00:10:00, 2019-01-01 00:15:00, 2019-01-01 00:20:00…etc
Dividing by 1000 to convert from ms to s as explained here
In my case, I had the epochs as a normal "column" (i.e. it wasn’t a TimeSeries).
I converted it as follows:
import pandas as pd
df['Timestamp'] = pd.to_datetime(df['Timestamp'] / 1000, unit='s')
Time was in milliseconds, so I needed to divide by a thousand. I set the unit='s' to mention that the Timestamp is in seconds (you can, however, use unit='ms' if you don’t want to divide by 1000, or depending on your original Timestamp unit).
A sample result
1673885284000 becomes 2023-01-16 4:08:04
from datetime import datetime
time = datetime.fromtimestamp(1676266245263 / 1000)
Example Output :
2023-02-13 05:30:45.263000
I have a pandas column of Timestamp data
In [27]: train["Original_Quote_Date"][6]
Out[27]: Timestamp('2013-12-25 00:00:00')
How can check equivalence of these objects to datetime.date objects of the type
datetime.date(2013, 12, 25)
You can convert a datetime.date object into a pandas Timestamp like this:
#!/usr/bin/env python3
# coding: utf-8
import pandas as pd
import datetime
# create a datetime data object
d_time = datetime.date(2010, 11, 12)
# create a pandas Timestamp object
t_stamp = pd.to_datetime('2010/11/12')
# cast `datetime_timestamp` as Timestamp object and compare
d_time2t_stamp = pd.to_datetime(d_time)
# print to double check
print(d_time)
print(t_stamp)
print(d_time2t_stamp)
# since the conversion succeds this prints `True`
print(d_time2t_stamp == t_stamp)
Use the .date method:
In [11]: t = pd.Timestamp('2013-12-25 00:00:00')
In [12]: t.date()
Out[12]: datetime.date(2013, 12, 25)
In [13]: t.date() == datetime.date(2013, 12, 25)
Out[13]: True
To compare against a DatetimeIndex (i.e. an array of Timestamps), you’ll want to do it the other way around:
In [21]: pd.Timestamp(datetime.date(2013, 12, 25))
Out[21]: Timestamp('2013-12-25 00:00:00')
In [22]: ts = pd.DatetimeIndex([t])
In [23]: ts == pd.Timestamp(datetime.date(2013, 12, 25))
Out[23]: array([ True], dtype=bool)
As of pandas 0.20.3, use .to_pydatetime() to convert any pandas.DateTimeIndex instances to Python datetime.datetime.
Assume time column is in timestamp integer msec format
1 day = 86400000 ms
Here you go:
day_divider = 86400000
df['time'] = df['time'].values.astype(dtype='datetime64[ms]') # for msec format
df['time'] = (df['time']/day_divider).values.astype(dtype='datetime64[D]') # for day format
So, got this from an IBM coursera tutorial.
data['date'] = data['TimeStamp'].apply(lambda d: datetime.date.fromtimestamp(d))
I’ve used the way recommended by Filomeno Gonzalez, albeit with a slight twist:
data['date'] = data['date'].apply(lambda x: x.date())
If I have a pandas DataFrame with timestamp column (1546300800000, 1546301100000, 1546301400000, 1546301700000, 1546302000000) and I want to convert this into date time format
import datetime
df['date'] = df['date'].apply(lambda x: datetime.datetime.fromtimestamp(x/1000.0))
This will return a column with the format 2019-01-01 00:00:00, 2019-01-01 00:05:00, 2019-01-01 00:10:00, 2019-01-01 00:15:00, 2019-01-01 00:20:00…etc
Dividing by 1000 to convert from ms to s as explained here
In my case, I had the epochs as a normal "column" (i.e. it wasn’t a TimeSeries).
I converted it as follows:
import pandas as pd
df['Timestamp'] = pd.to_datetime(df['Timestamp'] / 1000, unit='s')
Time was in milliseconds, so I needed to divide by a thousand. I set the unit='s' to mention that the Timestamp is in seconds (you can, however, use unit='ms' if you don’t want to divide by 1000, or depending on your original Timestamp unit).
A sample result
1673885284000 becomes 2023-01-16 4:08:04
from datetime import datetime
time = datetime.fromtimestamp(1676266245263 / 1000)
Example Output :
2023-02-13 05:30:45.263000