What is the most efficient way to store a list in the Django models?
Question:
Currently I have a lot of python objects in my code similar to the following:
class MyClass():
def __init__(self, name, friends):
self.myName = name
self.myFriends = [str(x) for x in friends]
Now I want to turn this into a Django model, where self.myName is a string field, and self.myFriends is a list of strings.
from django.db import models
class myDjangoModelClass():
myName = models.CharField(max_length=64)
myFriends = ??? # what goes here?
Since the list is such a common data structure in python, I sort of expected there to be a Django model field for it. I know I can use a ManyToMany or OneToMany relationship, but I was hoping to avoid that extra indirection in the code.
Edit:
I added this related question, which people may find useful.
Answers:
Would this relationship not be better expressed as a one-to-many foreign key relationship to a Friends
table? I understand that myFriends
are just strings but I would think that a better design would be to create a Friend
model and have MyClass
contain a foreign key realtionship to the resulting table.
Remember that this eventually has to end up in a relational database. So using relations really is the common way to solve this problem. If you absolutely insist on storing a list in the object itself, you could make it for example comma-separated, and store it in a string, and then provide accessor functions that split the string into a list. With that, you will be limited to a maximum number of strings, and you will lose efficient queries.
You can store virtually any object using a Django Pickle Field, ala this snippet:
class Course(models.Model):
name = models.CharField(max_length=256)
students = models.ManyToManyField(Student)
class Student(models.Model):
first_name = models.CharField(max_length=256)
student_number = models.CharField(max_length=128)
# other fields, etc...
friends = models.ManyToManyField('self')
“Premature optimization is the root of all evil.”
With that firmly in mind, let’s do this! Once your apps hit a certain point, denormalizing data is very common. Done correctly, it can save numerous expensive database lookups at the cost of a little more housekeeping.
To return a list
of friend names we’ll need to create a custom Django Field class that will return a list when accessed.
David Cramer posted a guide to creating a SeperatedValueField on his blog. Here is the code:
from django.db import models
class SeparatedValuesField(models.TextField):
__metaclass__ = models.SubfieldBase
def __init__(self, *args, **kwargs):
self.token = kwargs.pop('token', ',')
super(SeparatedValuesField, self).__init__(*args, **kwargs)
def to_python(self, value):
if not value: return
if isinstance(value, list):
return value
return value.split(self.token)
def get_db_prep_value(self, value):
if not value: return
assert(isinstance(value, list) or isinstance(value, tuple))
return self.token.join([unicode(s) for s in value])
def value_to_string(self, obj):
value = self._get_val_from_obj(obj)
return self.get_db_prep_value(value)
The logic of this code deals with serializing and deserializing values from the database to Python and vice versa. Now you can easily import and use our custom field in the model class:
from django.db import models
from custom.fields import SeparatedValuesField
class Person(models.Model):
name = models.CharField(max_length=64)
friends = SeparatedValuesField()
Using one-to-many relation (FK from Friend to parent class) will make your app more scalable (as you can trivially extend the Friend object with additional attributes beyond the simple name). And thus this is the best way
A simple way to store a list in Django is to just convert it into a JSON string, and then save that as Text in the model. You can then retrieve the list by converting the (JSON) string back into a python list. Here’s how:
The “list” would be stored in your Django model like so:
class MyModel(models.Model):
myList = models.TextField(null=True) # JSON-serialized (text) version of your list
In your view/controller code:
Storing the list in the database:
import simplejson as json # this would be just 'import json' in Python 2.7 and later
...
...
myModel = MyModel()
listIWantToStore = [1,2,3,4,5,'hello']
myModel.myList = json.dumps(listIWantToStore)
myModel.save()
Retrieving the list from the database:
jsonDec = json.decoder.JSONDecoder()
myPythonList = jsonDec.decode(myModel.myList)
Conceptually, here’s what’s going on:
>>> myList = [1,2,3,4,5,'hello']
>>> import simplejson as json
>>> myJsonList = json.dumps(myList)
>>> myJsonList
'[1, 2, 3, 4, 5, "hello"]'
>>> myJsonList.__class__
<type 'str'>
>>> jsonDec = json.decoder.JSONDecoder()
>>> myPythonList = jsonDec.decode(myJsonList)
>>> myPythonList
[1, 2, 3, 4, 5, u'hello']
>>> myPythonList.__class__
<type 'list'>
As this is an old question, and Django techniques must have changed significantly since, this answer reflects Django version 1.4, and is most likely applicable for v 1.5.
Django by default uses relational databases; you should make use of ’em. Map friendships to database relations (foreign key constraints) with the use of ManyToManyField. Doing so allows you to use RelatedManagers for friendlists, which use smart querysets. You can use all available methods such as filter
or values_list
.
Using ManyToManyField
relations and properties:
class MyDjangoClass(models.Model):
name = models.CharField(...)
friends = models.ManyToManyField("self")
@property
def friendlist(self):
# Watch for large querysets: it loads everything in memory
return list(self.friends.all())
You can access a user’s friend list this way:
joseph = MyDjangoClass.objects.get(name="Joseph")
friends_of_joseph = joseph.friendlist
Note however that these relations are symmetrical: if Joseph is a friend of Bob, then Bob is a friend of Joseph.
If you are using Django >= 1.9 with Postgres you can make use of ArrayField advantages
A field for storing lists of data. Most field types can be used, you
simply pass another field instance as the base_field. You may also
specify a size. ArrayField can be nested to store multi-dimensional
arrays.
It is also possible to nest array fields:
from django.contrib.postgres.fields import ArrayField
from django.db import models
class ChessBoard(models.Model):
board = ArrayField(
ArrayField(
models.CharField(max_length=10, blank=True),
size=8,
),
size=8,
)
As @thane-brimhall mentioned it is also possible to query elements directly. Documentation reference
In case you’re using postgres, you can use something like this:
class ChessBoard(models.Model):
board = ArrayField(
ArrayField(
models.CharField(max_length=10, blank=True),
size=8,
),
size=8,
)
if you need more details you can read in the link below:
https://docs.djangoproject.com/pt-br/1.9/ref/contrib/postgres/fields/
Storing a list of strings in Django model:
class Bar(models.Model):
foo = models.TextField(blank=True)
def set_list(self, element):
self.foo += "," + element if self.foo else element
def get_list(self):
return self.foo.split(",") if self.foo else None
and you can call it like this:
bars = Bar()
bars.set_list("str1")
bars.set_list("str2")
bar_list = bars.get_list()
for bar in bar_list:
print bar
Because in 2021 this post is first in google results.
This days exist nice elegant solution
from django.db.models import CharField, Model
from django_mysql.models import ListCharField
class Person(Model):
name = CharField()
post_nominals = ListCharField(
base_field=CharField(max_length=10),
size=6,
max_length=(6 * 11) # 6 * 10 character nominals, plus commas
)
Form
from django.db.models import IntegerField, Model
from django_mysql.models import ListTextField
class Widget(Model):
widget_group_ids = ListTextField(
base_field=IntegerField(),
size=100, # Maximum of 100 ids in list
)
Query
>>> Person.objects.create(name='Horatio', post_nominals=['PhD', 'Esq.', 'III'])
>>> Person.objects.create(name='Severus', post_nominals=['PhD', 'DPhil'])
>>> Person.objects.create(name='Paulus', post_nominals=[])
>>> Person.objects.filter(post_nominals__contains='PhD')
[<Person: Horatio>, <Person: Severus>]
>>> Person.objects.filter(post_nominals__contains='Esq.')
[<Person: Horatio>]
>>> Person.objects.filter(post_nominals__contains='DPhil')
[<Person: Severus>]
>>> Person.objects.filter(Q(post_nominals__contains='PhD') & Q(post_nominals__contains='III'))
[<Person: Horatio>]
As ListCharField is a subclass of CharField, any CharField options can be set too. Most importantly you’ll need to set max_length to determine how many characters to reserve in the database.
Example instantiation:
from django.db.models import CharField, Model
from django_mysql.models import ListCharField
class Person(Model):
name = CharField()
post_nominals = ListCharField(
base_field=CharField(max_length=10),
size=6,
max_length=(6 * 11), # 6 * 10 character nominals, plus commas
)
There are various answers addressing potential paths, such as ArrayField
if one is using PostgreSQL, TextField
by converting the list into a JSON string, creating a custom Django model field or just creating a new model and then using a ForeignKey as noted by Daniel Roseman here.
Still, one of my favourites (together with the ForeignKey option) for this use case was left out. More specifically, I’d like to reference the JSONField
as a better option. As per the documentation,
A field for storing JSON encoded data. In Python the data is represented in its Python native format: dictionaries, lists, strings, numbers, booleans and None.
JSONField is supported on MariaDB, MySQL 5.7.8+, Oracle, PostgreSQL, and SQLite (with the JSON1 extension enabled).
It has advantage over ArrayField
in that it is supported in more databases than PostgreSQL (since Django 3.1).
Also, it’s better than TextField
since this one was purposefully built to store JSON and requires less steps for operations like reading.
Also, it’s better than a custom field in that it comes already with Django (no need for extra work / maintenance).
Answer in 2023: Depends on the database you’re using with Django
This answer will cover PostgreSQL, MySQL, and SQLite.
PostgreSQL
In short: yes! PostgreSQL has an ArrayField
, and it’s simple. Code for models:
from django.db import models
from django.contrib.postgres.fields import ArrayField
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
myFriends = ArrayField(models.CharField(max_length=64)
Documentation for array fields in Django with PostgreSQL here.
MySQL
In short: yes! MySQL can also handle lists, and it is also simple. Code for models:
from django.db import models
from django_mysql.models import ListCharField
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
myFriends = ListCharField(base_field=CharField(max_length=64))
Documentation list fields in Django with MySQL here.
SQLite
In short: no. Django with SQLite does not support lists directly. You can use one of the hacks/workarounds in the other answers, or the JSON field also described in other answers, or you can do it "the right way" with foreign keys as described in the presently accepted answer. I would argue ManyToMany is the right way to go here rather than a ForeignKey since presumably multiple people could have the same friends. Code for models:
from django.db import models
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
class myFriend(models.model):
friend_name = models.CharField(max_length=64)
myName = models.ManyToManyField(myDjangoModelClass, related_name='myFriends')
Documentation on ManyToMany fields in Django with SQLite here.
Currently I have a lot of python objects in my code similar to the following:
class MyClass():
def __init__(self, name, friends):
self.myName = name
self.myFriends = [str(x) for x in friends]
Now I want to turn this into a Django model, where self.myName is a string field, and self.myFriends is a list of strings.
from django.db import models
class myDjangoModelClass():
myName = models.CharField(max_length=64)
myFriends = ??? # what goes here?
Since the list is such a common data structure in python, I sort of expected there to be a Django model field for it. I know I can use a ManyToMany or OneToMany relationship, but I was hoping to avoid that extra indirection in the code.
Edit:
I added this related question, which people may find useful.
Would this relationship not be better expressed as a one-to-many foreign key relationship to a Friends
table? I understand that myFriends
are just strings but I would think that a better design would be to create a Friend
model and have MyClass
contain a foreign key realtionship to the resulting table.
Remember that this eventually has to end up in a relational database. So using relations really is the common way to solve this problem. If you absolutely insist on storing a list in the object itself, you could make it for example comma-separated, and store it in a string, and then provide accessor functions that split the string into a list. With that, you will be limited to a maximum number of strings, and you will lose efficient queries.
You can store virtually any object using a Django Pickle Field, ala this snippet:
class Course(models.Model):
name = models.CharField(max_length=256)
students = models.ManyToManyField(Student)
class Student(models.Model):
first_name = models.CharField(max_length=256)
student_number = models.CharField(max_length=128)
# other fields, etc...
friends = models.ManyToManyField('self')
“Premature optimization is the root of all evil.”
With that firmly in mind, let’s do this! Once your apps hit a certain point, denormalizing data is very common. Done correctly, it can save numerous expensive database lookups at the cost of a little more housekeeping.
To return a list
of friend names we’ll need to create a custom Django Field class that will return a list when accessed.
David Cramer posted a guide to creating a SeperatedValueField on his blog. Here is the code:
from django.db import models
class SeparatedValuesField(models.TextField):
__metaclass__ = models.SubfieldBase
def __init__(self, *args, **kwargs):
self.token = kwargs.pop('token', ',')
super(SeparatedValuesField, self).__init__(*args, **kwargs)
def to_python(self, value):
if not value: return
if isinstance(value, list):
return value
return value.split(self.token)
def get_db_prep_value(self, value):
if not value: return
assert(isinstance(value, list) or isinstance(value, tuple))
return self.token.join([unicode(s) for s in value])
def value_to_string(self, obj):
value = self._get_val_from_obj(obj)
return self.get_db_prep_value(value)
The logic of this code deals with serializing and deserializing values from the database to Python and vice versa. Now you can easily import and use our custom field in the model class:
from django.db import models
from custom.fields import SeparatedValuesField
class Person(models.Model):
name = models.CharField(max_length=64)
friends = SeparatedValuesField()
Using one-to-many relation (FK from Friend to parent class) will make your app more scalable (as you can trivially extend the Friend object with additional attributes beyond the simple name). And thus this is the best way
A simple way to store a list in Django is to just convert it into a JSON string, and then save that as Text in the model. You can then retrieve the list by converting the (JSON) string back into a python list. Here’s how:
The “list” would be stored in your Django model like so:
class MyModel(models.Model):
myList = models.TextField(null=True) # JSON-serialized (text) version of your list
In your view/controller code:
Storing the list in the database:
import simplejson as json # this would be just 'import json' in Python 2.7 and later
...
...
myModel = MyModel()
listIWantToStore = [1,2,3,4,5,'hello']
myModel.myList = json.dumps(listIWantToStore)
myModel.save()
Retrieving the list from the database:
jsonDec = json.decoder.JSONDecoder()
myPythonList = jsonDec.decode(myModel.myList)
Conceptually, here’s what’s going on:
>>> myList = [1,2,3,4,5,'hello']
>>> import simplejson as json
>>> myJsonList = json.dumps(myList)
>>> myJsonList
'[1, 2, 3, 4, 5, "hello"]'
>>> myJsonList.__class__
<type 'str'>
>>> jsonDec = json.decoder.JSONDecoder()
>>> myPythonList = jsonDec.decode(myJsonList)
>>> myPythonList
[1, 2, 3, 4, 5, u'hello']
>>> myPythonList.__class__
<type 'list'>
As this is an old question, and Django techniques must have changed significantly since, this answer reflects Django version 1.4, and is most likely applicable for v 1.5.
Django by default uses relational databases; you should make use of ’em. Map friendships to database relations (foreign key constraints) with the use of ManyToManyField. Doing so allows you to use RelatedManagers for friendlists, which use smart querysets. You can use all available methods such as filter
or values_list
.
Using ManyToManyField
relations and properties:
class MyDjangoClass(models.Model):
name = models.CharField(...)
friends = models.ManyToManyField("self")
@property
def friendlist(self):
# Watch for large querysets: it loads everything in memory
return list(self.friends.all())
You can access a user’s friend list this way:
joseph = MyDjangoClass.objects.get(name="Joseph")
friends_of_joseph = joseph.friendlist
Note however that these relations are symmetrical: if Joseph is a friend of Bob, then Bob is a friend of Joseph.
If you are using Django >= 1.9 with Postgres you can make use of ArrayField advantages
A field for storing lists of data. Most field types can be used, you
simply pass another field instance as the base_field. You may also
specify a size. ArrayField can be nested to store multi-dimensional
arrays.
It is also possible to nest array fields:
from django.contrib.postgres.fields import ArrayField
from django.db import models
class ChessBoard(models.Model):
board = ArrayField(
ArrayField(
models.CharField(max_length=10, blank=True),
size=8,
),
size=8,
)
As @thane-brimhall mentioned it is also possible to query elements directly. Documentation reference
In case you’re using postgres, you can use something like this:
class ChessBoard(models.Model):
board = ArrayField(
ArrayField(
models.CharField(max_length=10, blank=True),
size=8,
),
size=8,
)
if you need more details you can read in the link below:
https://docs.djangoproject.com/pt-br/1.9/ref/contrib/postgres/fields/
Storing a list of strings in Django model:
class Bar(models.Model):
foo = models.TextField(blank=True)
def set_list(self, element):
self.foo += "," + element if self.foo else element
def get_list(self):
return self.foo.split(",") if self.foo else None
and you can call it like this:
bars = Bar()
bars.set_list("str1")
bars.set_list("str2")
bar_list = bars.get_list()
for bar in bar_list:
print bar
Because in 2021 this post is first in google results.
This days exist nice elegant solution
from django.db.models import CharField, Model
from django_mysql.models import ListCharField
class Person(Model):
name = CharField()
post_nominals = ListCharField(
base_field=CharField(max_length=10),
size=6,
max_length=(6 * 11) # 6 * 10 character nominals, plus commas
)
Form
from django.db.models import IntegerField, Model
from django_mysql.models import ListTextField
class Widget(Model):
widget_group_ids = ListTextField(
base_field=IntegerField(),
size=100, # Maximum of 100 ids in list
)
Query
>>> Person.objects.create(name='Horatio', post_nominals=['PhD', 'Esq.', 'III'])
>>> Person.objects.create(name='Severus', post_nominals=['PhD', 'DPhil'])
>>> Person.objects.create(name='Paulus', post_nominals=[])
>>> Person.objects.filter(post_nominals__contains='PhD')
[<Person: Horatio>, <Person: Severus>]
>>> Person.objects.filter(post_nominals__contains='Esq.')
[<Person: Horatio>]
>>> Person.objects.filter(post_nominals__contains='DPhil')
[<Person: Severus>]
>>> Person.objects.filter(Q(post_nominals__contains='PhD') & Q(post_nominals__contains='III'))
[<Person: Horatio>]
As ListCharField is a subclass of CharField, any CharField options can be set too. Most importantly you’ll need to set max_length to determine how many characters to reserve in the database.
Example instantiation:
from django.db.models import CharField, Model
from django_mysql.models import ListCharField
class Person(Model):
name = CharField()
post_nominals = ListCharField(
base_field=CharField(max_length=10),
size=6,
max_length=(6 * 11), # 6 * 10 character nominals, plus commas
)
There are various answers addressing potential paths, such as ArrayField
if one is using PostgreSQL, TextField
by converting the list into a JSON string, creating a custom Django model field or just creating a new model and then using a ForeignKey as noted by Daniel Roseman here.
Still, one of my favourites (together with the ForeignKey option) for this use case was left out. More specifically, I’d like to reference the JSONField
as a better option. As per the documentation,
A field for storing JSON encoded data. In Python the data is represented in its Python native format: dictionaries, lists, strings, numbers, booleans and None.
JSONField is supported on MariaDB, MySQL 5.7.8+, Oracle, PostgreSQL, and SQLite (with the JSON1 extension enabled).
It has advantage over ArrayField
in that it is supported in more databases than PostgreSQL (since Django 3.1).
Also, it’s better than TextField
since this one was purposefully built to store JSON and requires less steps for operations like reading.
Also, it’s better than a custom field in that it comes already with Django (no need for extra work / maintenance).
Answer in 2023: Depends on the database you’re using with Django
This answer will cover PostgreSQL, MySQL, and SQLite.
PostgreSQL
In short: yes! PostgreSQL has an ArrayField
, and it’s simple. Code for models:
from django.db import models
from django.contrib.postgres.fields import ArrayField
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
myFriends = ArrayField(models.CharField(max_length=64)
Documentation for array fields in Django with PostgreSQL here.
MySQL
In short: yes! MySQL can also handle lists, and it is also simple. Code for models:
from django.db import models
from django_mysql.models import ListCharField
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
myFriends = ListCharField(base_field=CharField(max_length=64))
Documentation list fields in Django with MySQL here.
SQLite
In short: no. Django with SQLite does not support lists directly. You can use one of the hacks/workarounds in the other answers, or the JSON field also described in other answers, or you can do it "the right way" with foreign keys as described in the presently accepted answer. I would argue ManyToMany is the right way to go here rather than a ForeignKey since presumably multiple people could have the same friends. Code for models:
from django.db import models
class myDjangoModelClass(models.Model):
myName = models.CharField(max_length=64)
class myFriend(models.model):
friend_name = models.CharField(max_length=64)
myName = models.ManyToManyField(myDjangoModelClass, related_name='myFriends')
Documentation on ManyToMany fields in Django with SQLite here.