How to convert datetime object to milliseconds
Question:
I am parsing datetime values as follows:
df['actualDateTime'] = pd.to_datetime(df['actualDateTime'])
How can I convert this datetime objects to milliseconds?
I didn’t see mention of milliseconds in the doc of to_datetime.
Update (Based on feedback):
This is the current version of the code that provides error TypeError: Cannot convert input to Timestamp. The column Date3 must contain milliseconds (as a numeric equivalent of a datetime object).
import pandas as pd
import time
s1 = {'Date' : ['2015-10-20T07:21:00.000','2015-10-19T07:18:00.000','2015-10-19T07:15:00.000']}
df = pd.DataFrame(s1)
df['Date2'] = pd.to_datetime(df['Date'])
t = pd.Timestamp(df['Date2'])
df['Date3'] = time.mktime(t.timetuple())
print df
Answers:
from datetime import datetime
print datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
>>>> OUTPUT >>>>
2015-11-02 18:04:32.926
pandas.to_datetime is to convert string or few other datatype to pandas datetime[ns]
In your instance initial 'actualDateTime' is not having milliseconds.So, if you are parsing a column which has milliseconds you will get data.
for example,
df
Out[60]:
a b
0 2015-11-02 18:04:32.926 0
1 2015-11-02 18:04:32.928 1
2 2015-11-02 18:04:32.927 2
df.a
Out[61]:
0 2015-11-02 18:04:32.926
1 2015-11-02 18:04:32.928
2 2015-11-02 18:04:32.927
Name: a, dtype: object
df.a = pd.to_datetime(df.a)
df.a
Out[63]:
0 2015-11-02 18:04:32.926
1 2015-11-02 18:04:32.928
2 2015-11-02 18:04:32.927
Name: a, dtype: datetime64[ns]
df.a.dt.nanosecond
Out[64]:
0 0
1 0
2 0
dtype: int64
df.a.dt.microsecond
Out[65]:
0 926000
1 928000
2 927000
dtype: int64
You can try pd.to_datetime(df['actualDateTime'], unit='ms')
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html
says this will denote in epoch, with variations ‘s’,’ms’, ‘ns’ …
Update
If you want in epoch timestamp of the form 14567899..
import pandas as pd
import time
t = pd.Timestamp('2015-10-19 07:22:00')
time.mktime(t.timetuple())
>> 1445219520.0
Latest update
df = pd.DataFrame(s1)
df1 = pd.to_datetime(df['Date'])
pd.DatetimeIndex(df1)
>>>DatetimeIndex(['2015-10-20 07:21:00', '2015-10-19 07:18:00',
'2015-10-19 07:15:00'],
dtype='datetime64[ns]', freq=None)
df1.astype(np.int64)
>>>0 1445325660000000000
1 1445239080000000000
2 1445238900000000000
df1.astype(np.int64) // 10**9
>>>0 1445325660
1 1445239080
2 1445238900
Name: Date, dtype: int64
Timestamps in pandas are always in nanoseconds.
This gives you milliseconds since the epoch (1970-01-01):
df['actualDateTime'] = df['actualDateTime'].astype(np.int64) / int(1e6)
For what it’s worth, to convert a single Pandas timestamp object to milliseconds, I had to do:
import time
time.mktime(<timestamp_object>.timetuple())*1000
This will return milliseconds from epoch
timestamp_object.timestamp() * 1000
For python >= 3.8, for e.g.
pd.DataFrame({'temp':[1,2,3]}, index = [pd.Timestamp.utcnow()]*3)
convert to milliseconds:
times = df.index.view(np.int64) // int(1e6)
print(times[0])
gives:
1666925409051
Note: to convert to seconds, similarly e.g.:
times = df.index.view(np.int64) // int(1e9)
print(times[0])
1666925409
I am parsing datetime values as follows:
df['actualDateTime'] = pd.to_datetime(df['actualDateTime'])
How can I convert this datetime objects to milliseconds?
I didn’t see mention of milliseconds in the doc of to_datetime.
Update (Based on feedback):
This is the current version of the code that provides error TypeError: Cannot convert input to Timestamp. The column Date3 must contain milliseconds (as a numeric equivalent of a datetime object).
import pandas as pd
import time
s1 = {'Date' : ['2015-10-20T07:21:00.000','2015-10-19T07:18:00.000','2015-10-19T07:15:00.000']}
df = pd.DataFrame(s1)
df['Date2'] = pd.to_datetime(df['Date'])
t = pd.Timestamp(df['Date2'])
df['Date3'] = time.mktime(t.timetuple())
print df
from datetime import datetime
print datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
>>>> OUTPUT >>>>
2015-11-02 18:04:32.926
pandas.to_datetime is to convert string or few other datatype to pandas datetime[ns]
In your instance initial 'actualDateTime' is not having milliseconds.So, if you are parsing a column which has milliseconds you will get data.
for example,
df
Out[60]:
a b
0 2015-11-02 18:04:32.926 0
1 2015-11-02 18:04:32.928 1
2 2015-11-02 18:04:32.927 2
df.a
Out[61]:
0 2015-11-02 18:04:32.926
1 2015-11-02 18:04:32.928
2 2015-11-02 18:04:32.927
Name: a, dtype: object
df.a = pd.to_datetime(df.a)
df.a
Out[63]:
0 2015-11-02 18:04:32.926
1 2015-11-02 18:04:32.928
2 2015-11-02 18:04:32.927
Name: a, dtype: datetime64[ns]
df.a.dt.nanosecond
Out[64]:
0 0
1 0
2 0
dtype: int64
df.a.dt.microsecond
Out[65]:
0 926000
1 928000
2 927000
dtype: int64
You can try pd.to_datetime(df['actualDateTime'], unit='ms')
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html
says this will denote in epoch, with variations ‘s’,’ms’, ‘ns’ …
Update
If you want in epoch timestamp of the form 14567899..
import pandas as pd
import time
t = pd.Timestamp('2015-10-19 07:22:00')
time.mktime(t.timetuple())
>> 1445219520.0
Latest update
df = pd.DataFrame(s1)
df1 = pd.to_datetime(df['Date'])
pd.DatetimeIndex(df1)
>>>DatetimeIndex(['2015-10-20 07:21:00', '2015-10-19 07:18:00',
'2015-10-19 07:15:00'],
dtype='datetime64[ns]', freq=None)
df1.astype(np.int64)
>>>0 1445325660000000000
1 1445239080000000000
2 1445238900000000000
df1.astype(np.int64) // 10**9
>>>0 1445325660
1 1445239080
2 1445238900
Name: Date, dtype: int64
Timestamps in pandas are always in nanoseconds.
This gives you milliseconds since the epoch (1970-01-01):
df['actualDateTime'] = df['actualDateTime'].astype(np.int64) / int(1e6)
For what it’s worth, to convert a single Pandas timestamp object to milliseconds, I had to do:
import time
time.mktime(<timestamp_object>.timetuple())*1000
This will return milliseconds from epoch
timestamp_object.timestamp() * 1000
For python >= 3.8, for e.g.
pd.DataFrame({'temp':[1,2,3]}, index = [pd.Timestamp.utcnow()]*3)
convert to milliseconds:
times = df.index.view(np.int64) // int(1e6)
print(times[0])
gives:
1666925409051
Note: to convert to seconds, similarly e.g.:
times = df.index.view(np.int64) // int(1e9)
print(times[0])
1666925409