Going from twitter date to Python datetime date
Question:
I am receiving twitter messages that are sent at a certain date in the following format from twitter:
Tue Mar 29 08:11:25 +0000 2011
I want to store these dates in ‘timestamp with time zone’ field in postgresql with djangos DateTimeField field. When I store that string however I get this error:
ValidationError: [u'Enter a valid date/time in YYYY-MM-DD HH:MM[:ss[.uuuuuu]] format.']
can I automatically transform the twitter datetype to a python datetime time (that does work elsewhere in my app for saving dates).
Answers:
you can convert the date using datetime.strptime(), or time.strptime(). however, those two functions cannot parse the timezone offset (see this bug).
so, the only solution i see is to split the date yourself, remove the timezone offset, feed the rest to strptime(), and process the offset manually…
have a look at this question, where you will find some hints on how to parse the offset yourself.
Give this a go. It assumes the date format from twitter is RFC822 compliant (see the question linked to by @Adrien).
A naive datetime object is constructed (i.e. no timezone info). It is adjusted according to the timezone offset to UTC. Unless you have a need to keep the original timezone, I’d store the date time as UTC and format to local time when you display it.
from datetime import datetime, timedelta
from email.utils import parsedate_tz
s = 'Tue Mar 29 08:11:25 +0000 2011'
def to_datetime(datestring):
time_tuple = parsedate_tz(datestring.strip())
dt = datetime(*time_tuple[:6])
return dt - timedelta(seconds=time_tuple[-1])
Writing something like this should convert a twitter date to a timestamp.
import time
ts = time.strftime('%Y-%m-%d %H:%M:%S', time.strptime(tweet['created_at'],'%a %b %d %H:%M:%S +0000 %Y'))
UPDATE
For Python 3, as per 2020, you can do it in this way:
from datetime import datetime
# dtime = tweet['created_at']
dtime = 'Fri Oct 09 10:01:41 +0000 2015'
new_datetime = datetime.strftime(datetime.strptime(dtime,'%a %b %d %H:%M:%S +0000 %Y'), '%Y-%m-%d %H:%M:%S')
print((new_datetime))
The following code will print a nice date (local time) from a Twitter date (UTC).
from datetime import datetime
from datetime import timezone
datetime.strptime(mydata["created_at"], '%a %b %d %H:%M:%S %z %Y').replace(
tzinfo=timezone.utc).astimezone(tz=None).strftime('%Y-%m-%d %H:%M:%S'))
Using a similar strategy as SoFolichon proposed, in Python 3.x you can also use pytz like:
from datetime import datetime, timezone
import pytz
datetime.strptime(tweets["created_at"], '%a %b %d %H:%M:%S %z %Y').replace(
tzinfo=timezone.utc).astimezone(pytz.timezone('US/Eastern')).strftime(
'%Y-%m-%d %H:%M:%S')
How about this? It doesn’t need any formatting strings.
import datetime
from email.utils import mktime_tz, parsedate_tz
def parse_datetime(value):
time_tuple = parsedate_tz(value)
timestamp = mktime_tz(time_tuple)
return datetime.datetime.fromtimestamp(timestamp)
print(parse_datetime('Tue Mar 29 08:11:25 +0000 2011'))
#2011-03-29 10:11:25
My system is at GMT +2 hence the difference included.
A little bit old but using parse really help me with this issue
from datetime import datetime
from dateutil.parser import parse
date = 'Fri May 10 00:44:04 +0000 2019'
dt = parse(date)
print(dt)
# 2019-05-10 00:44:04+00:00
To get datetime with timezone you can simple use datetime.strptime as follow:
from datetime import datetime
s = 'Wed Jun 05 05:34:02 +0000 2019'
created_at = datetime.strptime(s, '%a %b %d %H:%M:%S %z %Y')
print(created_at)
#2019-06-05 05:34:02+00:00
The initial problem I was having was converting from the datetime that the twitter api gives to String.
The following works which addresses different comments people seem to have for above solutions which are a little unclear as to whether the starting date is already in string format or not.
This works for Python 2.7
With a tweet from the API, tweet.created_at gives the date in datetime format. At the top of your file, add from datetime import datetime
then use the following to get the corresponding string.
datetime.strftime(tweet.created_at,'%a %b %d %H:%M:%S %z %Y').
You can then use this string as described in other comments to manipulate it.
Twitter API V2 sends date strings that look like this:
2020-12-15T20:17:10.000Z
This worked, to convert from string to datetime:
datetime.datetime.strptime(THE_STRING,"%Y-%m-%dT%H:%M:%S.%fZ")
The end looks like a timezone, but it’s milliseconds, hence the %f. The final character, "Z" is a timezone code that means UTC, as explained, here.
I am receiving twitter messages that are sent at a certain date in the following format from twitter:
Tue Mar 29 08:11:25 +0000 2011
I want to store these dates in ‘timestamp with time zone’ field in postgresql with djangos DateTimeField field. When I store that string however I get this error:
ValidationError: [u'Enter a valid date/time in YYYY-MM-DD HH:MM[:ss[.uuuuuu]] format.']
can I automatically transform the twitter datetype to a python datetime time (that does work elsewhere in my app for saving dates).
you can convert the date using datetime.strptime(), or time.strptime(). however, those two functions cannot parse the timezone offset (see this bug).
so, the only solution i see is to split the date yourself, remove the timezone offset, feed the rest to strptime(), and process the offset manually…
have a look at this question, where you will find some hints on how to parse the offset yourself.
Give this a go. It assumes the date format from twitter is RFC822 compliant (see the question linked to by @Adrien).
A naive datetime object is constructed (i.e. no timezone info). It is adjusted according to the timezone offset to UTC. Unless you have a need to keep the original timezone, I’d store the date time as UTC and format to local time when you display it.
from datetime import datetime, timedelta
from email.utils import parsedate_tz
s = 'Tue Mar 29 08:11:25 +0000 2011'
def to_datetime(datestring):
time_tuple = parsedate_tz(datestring.strip())
dt = datetime(*time_tuple[:6])
return dt - timedelta(seconds=time_tuple[-1])
Writing something like this should convert a twitter date to a timestamp.
import time
ts = time.strftime('%Y-%m-%d %H:%M:%S', time.strptime(tweet['created_at'],'%a %b %d %H:%M:%S +0000 %Y'))
UPDATE
For Python 3, as per 2020, you can do it in this way:
from datetime import datetime
# dtime = tweet['created_at']
dtime = 'Fri Oct 09 10:01:41 +0000 2015'
new_datetime = datetime.strftime(datetime.strptime(dtime,'%a %b %d %H:%M:%S +0000 %Y'), '%Y-%m-%d %H:%M:%S')
print((new_datetime))
The following code will print a nice date (local time) from a Twitter date (UTC).
from datetime import datetime
from datetime import timezone
datetime.strptime(mydata["created_at"], '%a %b %d %H:%M:%S %z %Y').replace(
tzinfo=timezone.utc).astimezone(tz=None).strftime('%Y-%m-%d %H:%M:%S'))
Using a similar strategy as SoFolichon proposed, in Python 3.x you can also use pytz like:
from datetime import datetime, timezone
import pytz
datetime.strptime(tweets["created_at"], '%a %b %d %H:%M:%S %z %Y').replace(
tzinfo=timezone.utc).astimezone(pytz.timezone('US/Eastern')).strftime(
'%Y-%m-%d %H:%M:%S')
How about this? It doesn’t need any formatting strings.
import datetime
from email.utils import mktime_tz, parsedate_tz
def parse_datetime(value):
time_tuple = parsedate_tz(value)
timestamp = mktime_tz(time_tuple)
return datetime.datetime.fromtimestamp(timestamp)
print(parse_datetime('Tue Mar 29 08:11:25 +0000 2011'))
#2011-03-29 10:11:25
My system is at GMT +2 hence the difference included.
A little bit old but using parse really help me with this issue
from datetime import datetime
from dateutil.parser import parse
date = 'Fri May 10 00:44:04 +0000 2019'
dt = parse(date)
print(dt)
# 2019-05-10 00:44:04+00:00
To get datetime with timezone you can simple use datetime.strptime as follow:
from datetime import datetime
s = 'Wed Jun 05 05:34:02 +0000 2019'
created_at = datetime.strptime(s, '%a %b %d %H:%M:%S %z %Y')
print(created_at)
#2019-06-05 05:34:02+00:00
The initial problem I was having was converting from the datetime that the twitter api gives to String.
The following works which addresses different comments people seem to have for above solutions which are a little unclear as to whether the starting date is already in string format or not.
This works for Python 2.7
With a tweet from the API, tweet.created_at gives the date in datetime format. At the top of your file, add from datetime import datetime
then use the following to get the corresponding string.
datetime.strftime(tweet.created_at,'%a %b %d %H:%M:%S %z %Y').
You can then use this string as described in other comments to manipulate it.
Twitter API V2 sends date strings that look like this:
2020-12-15T20:17:10.000Z
This worked, to convert from string to datetime:
datetime.datetime.strptime(THE_STRING,"%Y-%m-%dT%H:%M:%S.%fZ")
The end looks like a timezone, but it’s milliseconds, hence the %f. The final character, "Z" is a timezone code that means UTC, as explained, here.