Always Defer a Field in Django
Question:
How do I make a field on a Django model deferred for all queries of that model without needing to put a defer on every query?
Research
This was requested as a feature in 2014 and rejected in 2022.
Baring such a feature native to Django, the obvious idea is to make a custom manager like this:
class DeferedFieldManager(models.Manager):
def __init__(self, defered_fields=[]):
super().__init__()
self.defered_fields = defered_fields
def get_queryset(self, *args, **kwargs):
return super().get_queryset(*args, **kwargs
).defer(*self.defered_fields)
class B(models.Model):
pass
class A(models.Model):
big_field = models.TextField(null=True)
b = models.ForeignKey(B, related_name="a_s")
objects = DeferedFieldManager(["big_field"])
class C(models.Model):
a = models.ForeignKey(A)
class D(models.Model):
a = models.OneToOneField(A)
class E(models.Model):
a_s = models.ManyToManyField(A)
However, while this works for A.objects.first()
(direct lookups), it doesn’t work for B.objects.first().a_s.all()
(one-to-manys), C.objects.first().a
(many-to-ones), D.objects.first().a
(one-to-ones), or E.objects.first().a_s.all()
(many-to-manys).
The thing I find particularly confusing here is that this is the default manager for my object, which means it should also be the default for the reverse lookups (the one-to-manys and many-to-manys), yet this isn’t working. Per the Django docs:
By default the RelatedManager used for reverse relations is a subclass of the default manager for that model.
An easy way to test this is to drop the field that should be deferred from the database, and the code will only error with an OperationalError: no such column
if the field is not properly deferred. To test, do the following steps:
- Data setup:
b = B.objects.create()
a = A.objects.create(b=b)
c = C.objects.create(a=a)
d = D.objects.create(a=a)
e = E.objects.create()
e.a_s.add(a)
- Comment out
big_field
manage.py makemigrations
manage.py migrate
- Comment in
big_field
- Run tests:
from django.db import OperationalError
def test(test_name, f, attr=None):
try:
if attr:
x = getattr(f(), attr)
else:
x = f()
assert isinstance(x, A)
print(f"{test_name}:tpass")
except OperationalError:
print(f"{test_name}:tFAIL!!!")
test("Direct Lookup", A.objects.first)
test("One-to-Many", B.objects.first().a_s.first)
test("Many-to-One", C.objects.first, "a")
test("One-to-One", D.objects.first, "a")
test("Many-to-Many", E.objects.first().a_s.first)
If the tests above all pass, the field has been properly deferred.
I’m currently getting:
Direct Lookup: pass
One-to-Many: FAIL!!!
Many-to-One: FAIL!!!
One-to-One: FAIL!!!
Many-to-Many: FAIL!!!
Partial Answer
@aaron’s answer solves half of the failing cases.
If I change A
to have:
class Meta:
base_manager_name = 'objects'
I now get the following from tests:
Direct Lookup: pass
One-to-Many: FAIL!!!
Many-to-One: pass
One-to-One: pass
Many-to-Many: FAIL!!!
This still does not work for the revere lookups.
Answers:
This is works on Django 1.4.22, i use it in admin-panel, to made the answer from DB smaller:
C.objects.select_related('a').defer('a__some_field').first()
And it should works too (not tested):
D.objects.select_related('a').defer('a__some_field').first()
We have created custom queryset methods, in your case:
class C_obj_queryset(QuerySet):
def get_deferred_for_related_model(self, model):
return self.select_related(model).defer(*model.objects.defered_fields)
more here:
https://docs.djangoproject.com/en/4.1/ref/models/querysets/#defer
You can defer loading of fields in related models (if the related models are loading via select_related()) by using the standard
double-underscore notation to separate related fields.
Set Meta.base_manager_name
to 'objects'
.
class A(models.Model):
big_field = models.TextField(null=True)
b = models.ForeignKey(B, related_name="a_s")
objects = DeferedFieldManager(["big_field"])
class Meta:
base_manager_name = 'objects'
From https://docs.djangoproject.com/en/4.1/topics/db/managers/#using-managers-for-related-object-access:
Using managers for related object access
By default, Django uses an instance of the Model._base_manager
manager class when accessing related objects (i.e. choice.question
), not the _default_manager
on the related object. This is because Django needs to be able to retrieve the related object, even if it would otherwise be filtered out (and hence be inaccessible) by the default manager.
If the normal base manager class (django.db.models.Manager
) isn’t appropriate for your circumstances, you can tell Django which class to use by setting Meta.base_manager_name
.
Reverse Many-to-One and Many-to-Many managers
The "One-To-Many" case in the question is a Reverse Many-To-One.
Django subclasses the manager class to override the behaviour, and then instantiates it — without the defered_fields
argument passed to __init__
since
django.db.models.Manager
and its subclasses are not expected to have parameters.
Thus, you need something like:
def make_defered_field_manager(defered_fields):
class DeferedFieldManager(models.Manager):
def get_queryset(self, *args, **kwargs):
return super().get_queryset(*args, **kwargs).defer(*defered_fields)
return DeferedFieldManager()
Usage:
# objects = DeferedFieldManager(["big_field"])
objects = make_defered_field_manager(["big_field"])
How do I make a field on a Django model deferred for all queries of that model without needing to put a defer on every query?
Research
This was requested as a feature in 2014 and rejected in 2022.
Baring such a feature native to Django, the obvious idea is to make a custom manager like this:
class DeferedFieldManager(models.Manager):
def __init__(self, defered_fields=[]):
super().__init__()
self.defered_fields = defered_fields
def get_queryset(self, *args, **kwargs):
return super().get_queryset(*args, **kwargs
).defer(*self.defered_fields)
class B(models.Model):
pass
class A(models.Model):
big_field = models.TextField(null=True)
b = models.ForeignKey(B, related_name="a_s")
objects = DeferedFieldManager(["big_field"])
class C(models.Model):
a = models.ForeignKey(A)
class D(models.Model):
a = models.OneToOneField(A)
class E(models.Model):
a_s = models.ManyToManyField(A)
However, while this works for A.objects.first()
(direct lookups), it doesn’t work for B.objects.first().a_s.all()
(one-to-manys), C.objects.first().a
(many-to-ones), D.objects.first().a
(one-to-ones), or E.objects.first().a_s.all()
(many-to-manys).
The thing I find particularly confusing here is that this is the default manager for my object, which means it should also be the default for the reverse lookups (the one-to-manys and many-to-manys), yet this isn’t working. Per the Django docs:
By default the RelatedManager used for reverse relations is a subclass of the default manager for that model.
An easy way to test this is to drop the field that should be deferred from the database, and the code will only error with an OperationalError: no such column
if the field is not properly deferred. To test, do the following steps:
- Data setup:
b = B.objects.create() a = A.objects.create(b=b) c = C.objects.create(a=a) d = D.objects.create(a=a) e = E.objects.create() e.a_s.add(a)
- Comment out
big_field
manage.py makemigrations
manage.py migrate
- Comment in
big_field
- Run tests:
from django.db import OperationalError def test(test_name, f, attr=None): try: if attr: x = getattr(f(), attr) else: x = f() assert isinstance(x, A) print(f"{test_name}:tpass") except OperationalError: print(f"{test_name}:tFAIL!!!") test("Direct Lookup", A.objects.first) test("One-to-Many", B.objects.first().a_s.first) test("Many-to-One", C.objects.first, "a") test("One-to-One", D.objects.first, "a") test("Many-to-Many", E.objects.first().a_s.first)
If the tests above all pass, the field has been properly deferred.
I’m currently getting:
Direct Lookup: pass
One-to-Many: FAIL!!!
Many-to-One: FAIL!!!
One-to-One: FAIL!!!
Many-to-Many: FAIL!!!
Partial Answer
@aaron’s answer solves half of the failing cases.
If I change A
to have:
class Meta:
base_manager_name = 'objects'
I now get the following from tests:
Direct Lookup: pass
One-to-Many: FAIL!!!
Many-to-One: pass
One-to-One: pass
Many-to-Many: FAIL!!!
This still does not work for the revere lookups.
This is works on Django 1.4.22, i use it in admin-panel, to made the answer from DB smaller:
C.objects.select_related('a').defer('a__some_field').first()
And it should works too (not tested):
D.objects.select_related('a').defer('a__some_field').first()
We have created custom queryset methods, in your case:
class C_obj_queryset(QuerySet):
def get_deferred_for_related_model(self, model):
return self.select_related(model).defer(*model.objects.defered_fields)
more here:
https://docs.djangoproject.com/en/4.1/ref/models/querysets/#defer
You can defer loading of fields in related models (if the related models are loading via select_related()) by using the standard
double-underscore notation to separate related fields.
Set Meta.base_manager_name
to 'objects'
.
class A(models.Model):
big_field = models.TextField(null=True)
b = models.ForeignKey(B, related_name="a_s")
objects = DeferedFieldManager(["big_field"])
class Meta:
base_manager_name = 'objects'
From https://docs.djangoproject.com/en/4.1/topics/db/managers/#using-managers-for-related-object-access:
Using managers for related object access
By default, Django uses an instance of the
Model._base_manager
manager class when accessing related objects (i.e.choice.question
), not the_default_manager
on the related object. This is because Django needs to be able to retrieve the related object, even if it would otherwise be filtered out (and hence be inaccessible) by the default manager.If the normal base manager class (
django.db.models.Manager
) isn’t appropriate for your circumstances, you can tell Django which class to use by settingMeta.base_manager_name
.
Reverse Many-to-One and Many-to-Many managers
The "One-To-Many" case in the question is a Reverse Many-To-One.
Django subclasses the manager class to override the behaviour, and then instantiates it — without the defered_fields
argument passed to __init__
since
django.db.models.Manager
and its subclasses are not expected to have parameters.
Thus, you need something like:
def make_defered_field_manager(defered_fields):
class DeferedFieldManager(models.Manager):
def get_queryset(self, *args, **kwargs):
return super().get_queryset(*args, **kwargs).defer(*defered_fields)
return DeferedFieldManager()
Usage:
# objects = DeferedFieldManager(["big_field"])
objects = make_defered_field_manager(["big_field"])