Python/Django – Expected a string or bytes-like object
Question:
I’ve looked around, and I’ve seen a couple of solutions that I thought would work for my own code, but seemed to do nothing.
I am trying to pull data from a JSON file and add the information to arrays. The arrays will then be used to input data into several objects that I add to the database (I know, this is terribly inefficient, but the way I wrote the original script, I didn’t have to add objects to a database. I am planning on changing that).
Object contains a date time associated with it. I used Python’s strptime function to convert the strings into datetime objects.
In my models.py, I have a DateTimeField for the datetime.
When I try and migrate, however, I get the following error:
TypeError: expected string or bytes-like object
Here is the entire traceback:
Applying interactive_table.0016_auto_20161024_2259...Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/operations/fields.py", line 84, in database_forwards
field,
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/sqlite3/schema.py", line 231, in add_field
self._remake_table(model, create_fields=[field])
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/sqlite3/schema.py", line 113, in _remake_table
self.effective_default(field)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 221, in effective_default
default = field.get_db_prep_save(default, self.connection)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 755, in get_db_prep_save
prepared=False)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1438, in get_db_prep_value
value = self.get_prep_value(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1417, in get_prep_value
value = super(DateTimeField, self).get_prep_value(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1275, in get_prep_value
return self.to_python(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1378, in to_python
parsed = parse_datetime(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/utils/dateparse.py", line 93, in parse_datetime
match = datetime_re.match(value)
For reference, here is my views.py:
def getInfo(counter, hekJSON):
for counter in range(len(hekJSON["Events"])):
dateAndTimeHEK.append(convertTimes(hekJSON["Events"][counter]["startTime"]))
xcen.append(float("%.2f" % hekJSON["Events"][counter]["xCen"]))
ycen.append(float("%.2f" % hekJSON["Events"][counter]["yCen"]))
xfov.append(float("%.2f" % hekJSON["Events"][counter]["raster_fovx"]))
yfov.append(float("%.2f" % hekJSON["Events"][counter]["raster_fovy"]))
sciObj.append(hekJSON["Events"][counter]["sciObjectives"])
def convertTimes(dateAndTime):
dateTime = datetime.datetime.strptime(dateAndTime, '%Y-%m-%d %H:%M:%S')
return dateTime
def display(request):
noaaNmbr='11809'
#for right now, this is the only active region that I'm pulling data for. When I get the graph up and running, I will make it more interactive for the user so that he can
urlData = "http://www.lmsal.com/hek/hcr?cmd=search-events3&outputformat=json&instrument=IRIS&noaanum="+ noaaNmbr +"&hasData=true"
webUrl = urlopen(urlData)
counter = 0
data = webUrl.read().decode('utf-8')
hekJSON = json.loads(data)
getInfo(counter, hekJSON)
for i in range(getNumberOfEntries(hekJSON)):
observation = models.HEK_Observations(noaaNmbr=noaaNmbr, dateAndTime=dateAndTimeHEK[i], xcen=xcen[i], ycen=ycen[i], xfov=xfov[i], yfov=yfov[i], sciObj=sciObj[i])
observation.save()
return render(request, 'template.html', {'obj': models.HEK_Observations.objects.filter(noaaNmbr=noaaNmbr)})
Here is my models.py:
from django.db import models
import datetime
class HEK_Observations(models.Model):
dateAndTime = models.DateTimeField(max_length = 40, default = None, verbose_name = 'Date And Time')
Here is my urls.py:
from django.conf.urls import url
from . import views
urlpatterns = [
# /table/
url(r'^$', views.display, name='display'),
]
Other solutions to similar problems have said that the problem lies with my default value. However, when I set no defaults (like shown in the code), I still get the same error.
Answers:
You’re getting hte error on the migration, so I’m guessing your datetime field was a string, and you’ve made it a date time field. The problem seems to be how your migration is going from those text-elements into dates. Potentially you need to scrub your db either dumping it (if its test data) or making a better manual migration script.
Make Migrations isn’t perfect – if you have existing data thats in different formats you have to make the code to modify the data into new formats and preserve information.
If its just test data (as it usually is), you can just truncate the table – either manually now, (delete the database even), or with a line in the migration script.
Its bad form to ‘delete’ your migration files and make them again (especially if you have multiple users), but if you have good processes around controlling it, it can be done to make clean releases (i.e. one migration per release).
You should Delete migration files and run migrations again
JUST IN CASE SOMEONE RUNS INTO THIS ERROR OF RECENT.
STEP 1: Go to the migrations folder "migrations".
STEP 2: Navigate to the migration file, the recent one, for example, "0008_auto_20210821_0912.py".
STEP 3: Change the default value that is there from whatever value to "2021-01-05" for DateField or "2021-01-05 06:00:00.000000-09:00" for DateTimeField
I could not agree less with all of the answers provided. The Error raised is perfectly valid. The real issue here is that for your model you have set default=None for the field dateAndTime.
None is not of type DateTime. So when you run your migration, this will raise a TypeError. To fix, delete the migration that was created, remove default = None and instead use blank=True and null=True,
as follows:
class HEKObservations(models.Model):
date_and_time = models.DateTimeField(max_length=40,
verbose_name='Date And Time',
blank=True,
null=True)
The latter will allow the field to remain empty.
P.S. also note the Python naming convention used.
I’ve looked around, and I’ve seen a couple of solutions that I thought would work for my own code, but seemed to do nothing.
I am trying to pull data from a JSON file and add the information to arrays. The arrays will then be used to input data into several objects that I add to the database (I know, this is terribly inefficient, but the way I wrote the original script, I didn’t have to add objects to a database. I am planning on changing that).
Object contains a date time associated with it. I used Python’s strptime function to convert the strings into datetime objects.
In my models.py, I have a DateTimeField for the datetime.
When I try and migrate, however, I get the following error:
TypeError: expected string or bytes-like object
Here is the entire traceback:
Applying interactive_table.0016_auto_20161024_2259...Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/migrations/operations/fields.py", line 84, in database_forwards
field,
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/sqlite3/schema.py", line 231, in add_field
self._remake_table(model, create_fields=[field])
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/sqlite3/schema.py", line 113, in _remake_table
self.effective_default(field)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 221, in effective_default
default = field.get_db_prep_save(default, self.connection)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 755, in get_db_prep_save
prepared=False)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1438, in get_db_prep_value
value = self.get_prep_value(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1417, in get_prep_value
value = super(DateTimeField, self).get_prep_value(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1275, in get_prep_value
return self.to_python(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 1378, in to_python
parsed = parse_datetime(value)
File "/Users/andrewho/anaconda/lib/python3.5/site-packages/django/utils/dateparse.py", line 93, in parse_datetime
match = datetime_re.match(value)
For reference, here is my views.py:
def getInfo(counter, hekJSON):
for counter in range(len(hekJSON["Events"])):
dateAndTimeHEK.append(convertTimes(hekJSON["Events"][counter]["startTime"]))
xcen.append(float("%.2f" % hekJSON["Events"][counter]["xCen"]))
ycen.append(float("%.2f" % hekJSON["Events"][counter]["yCen"]))
xfov.append(float("%.2f" % hekJSON["Events"][counter]["raster_fovx"]))
yfov.append(float("%.2f" % hekJSON["Events"][counter]["raster_fovy"]))
sciObj.append(hekJSON["Events"][counter]["sciObjectives"])
def convertTimes(dateAndTime):
dateTime = datetime.datetime.strptime(dateAndTime, '%Y-%m-%d %H:%M:%S')
return dateTime
def display(request):
noaaNmbr='11809'
#for right now, this is the only active region that I'm pulling data for. When I get the graph up and running, I will make it more interactive for the user so that he can
urlData = "http://www.lmsal.com/hek/hcr?cmd=search-events3&outputformat=json&instrument=IRIS&noaanum="+ noaaNmbr +"&hasData=true"
webUrl = urlopen(urlData)
counter = 0
data = webUrl.read().decode('utf-8')
hekJSON = json.loads(data)
getInfo(counter, hekJSON)
for i in range(getNumberOfEntries(hekJSON)):
observation = models.HEK_Observations(noaaNmbr=noaaNmbr, dateAndTime=dateAndTimeHEK[i], xcen=xcen[i], ycen=ycen[i], xfov=xfov[i], yfov=yfov[i], sciObj=sciObj[i])
observation.save()
return render(request, 'template.html', {'obj': models.HEK_Observations.objects.filter(noaaNmbr=noaaNmbr)})
Here is my models.py:
from django.db import models
import datetime
class HEK_Observations(models.Model):
dateAndTime = models.DateTimeField(max_length = 40, default = None, verbose_name = 'Date And Time')
Here is my urls.py:
from django.conf.urls import url
from . import views
urlpatterns = [
# /table/
url(r'^$', views.display, name='display'),
]
Other solutions to similar problems have said that the problem lies with my default value. However, when I set no defaults (like shown in the code), I still get the same error.
You’re getting hte error on the migration, so I’m guessing your datetime field was a string, and you’ve made it a date time field. The problem seems to be how your migration is going from those text-elements into dates. Potentially you need to scrub your db either dumping it (if its test data) or making a better manual migration script.
Make Migrations isn’t perfect – if you have existing data thats in different formats you have to make the code to modify the data into new formats and preserve information.
If its just test data (as it usually is), you can just truncate the table – either manually now, (delete the database even), or with a line in the migration script.
Its bad form to ‘delete’ your migration files and make them again (especially if you have multiple users), but if you have good processes around controlling it, it can be done to make clean releases (i.e. one migration per release).
You should Delete migration files and run migrations again
JUST IN CASE SOMEONE RUNS INTO THIS ERROR OF RECENT.
STEP 1: Go to the migrations folder "migrations".
STEP 2: Navigate to the migration file, the recent one, for example, "0008_auto_20210821_0912.py".
STEP 3: Change the default value that is there from whatever value to "2021-01-05" for DateField or "2021-01-05 06:00:00.000000-09:00" for DateTimeField
I could not agree less with all of the answers provided. The Error raised is perfectly valid. The real issue here is that for your model you have set default=None for the field dateAndTime.
None is not of type DateTime. So when you run your migration, this will raise a TypeError. To fix, delete the migration that was created, remove default = None and instead use blank=True and null=True,
as follows:
class HEKObservations(models.Model):
date_and_time = models.DateTimeField(max_length=40,
verbose_name='Date And Time',
blank=True,
null=True)
The latter will allow the field to remain empty.
P.S. also note the Python naming convention used.