Is django prefetch_related supposed to work with GenericRelation

UPDATE 2022: The original ticked #24272 which I opened 8 years ago about this issue is now closed in favor of #33651, which once implemented will give us a new syntax to do this type of prefetches.

============== END OF UPDATE ==============

What’s all about?

Django has a GenericRelation class, which adds a “reverse” generic relationship to enable an additional API.

It turns out we can use this reverse-generic-relation for filtering or ordering, but we can’t use it inside prefetch_related.

I was wondering if this is a bug, or its not supposed to work, or its something that can be implemented in the feature.

Let me show you with some examples what I mean.

Lets say we have two main models: Movies and Books.

• Movies have a Director
• Books have an Author

And we want to assign tags to our Movies and Books, but instead of using MovieTag and BookTag models, we want to use a single TaggedItem class with a GFK to Movie or Book.

Here is the model structure:

from django.db import models
from django.contrib.contenttypes.fields import GenericForeignKey, GenericRelation
from django.contrib.contenttypes.models import ContentType


class TaggedItem(models.Model):
    tag = models.SlugField()
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = GenericForeignKey('content_type', 'object_id')

    def __unicode__(self):
        return self.tag


class Director(models.Model):
    name = models.CharField(max_length=100)

    def __unicode__(self):
        return self.name


class Movie(models.Model):
    name = models.CharField(max_length=100)
    director = models.ForeignKey(Director)
    tags = GenericRelation(TaggedItem, related_query_name='movies')

    def __unicode__(self):
        return self.name


class Author(models.Model):
    name = models.CharField(max_length=100)

    def __unicode__(self):
        return self.name


class Book(models.Model):
    name = models.CharField(max_length=100)
    author = models.ForeignKey(Author)
    tags = GenericRelation(TaggedItem, related_query_name='books')

    def __unicode__(self):
        return self.name

And some initial data:

>>> from tags.models import Book, Movie, Author, Director, TaggedItem
>>> a = Author.objects.create(name='E L James')
>>> b1 = Book.objects.create(name='Fifty Shades of Grey', author=a)
>>> b2 = Book.objects.create(name='Fifty Shades Darker', author=a)
>>> b3 = Book.objects.create(name='Fifty Shades Freed', author=a)
>>> d = Director.objects.create(name='James Gunn')
>>> m1 = Movie.objects.create(name='Guardians of the Galaxy', director=d)
>>> t1 = TaggedItem.objects.create(content_object=b1, tag='roman')
>>> t2 = TaggedItem.objects.create(content_object=b2, tag='roman')
>>> t3 = TaggedItem.objects.create(content_object=b3, tag='roman')
>>> t4 = TaggedItem.objects.create(content_object=m1, tag='action movie')

So as the docs show we can do stuff like this.

>>> b1.tags.all()
[<TaggedItem: roman>]
>>> m1.tags.all()
[<TaggedItem: action movie>]
>>> TaggedItem.objects.filter(books__author__name='E L James')
[<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>]
>>> TaggedItem.objects.filter(movies__director__name='James Gunn')
[<TaggedItem: action movie>]
>>> Book.objects.all().prefetch_related('tags')
[<Book: Fifty Shades of Grey>, <Book: Fifty Shades Darker>, <Book: Fifty Shades Freed>]
>>> Book.objects.filter(tags__tag='roman')
[<Book: Fifty Shades of Grey>, <Book: Fifty Shades Darker>, <Book: Fifty Shades Freed>]

But, if we try to prefetch some related data of TaggedItem via this reverse generic relation, we are going to get an AttributeError.

>>> TaggedItem.objects.all().prefetch_related('books')
Traceback (most recent call last):
  ...
AttributeError: 'Book' object has no attribute 'object_id'

Some of you may ask, why I just don’t use content_object instead of books here? The reason is, because this only work when we want to:

1. prefetch only one level deep from querysets containing different type of content_object.

   >>> TaggedItem.objects.all().prefetch_related('content_object')
   [<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: action movie>]
   

2. prefetch many levels but from querysets containing only one type of content_object.

   >>> TaggedItem.objects.filter(books__author__name='E L James').prefetch_related('content_object__author')
   [<TaggedItem: roman>, <TaggedItem: roman>, <TaggedItem: roman>]
   

But, if we want both 1) and 2) (to prefetch many levels from queryset containing different types of content_objects, we can’t use content_object.

>>> TaggedItem.objects.all().prefetch_related('content_object__author')
Traceback (most recent call last):
  ...
AttributeError: 'Movie' object has no attribute 'author_id'

Django thinks that all content_objects are Books, and thus they have an Author.

Now imagine the situation where we want to prefetch not only the books with their author, but also the movies with their director. Here are few attempts.

The silly way:

>>> TaggedItem.objects.all().prefetch_related(
...     'content_object__author',
...     'content_object__director',
... )
Traceback (most recent call last):
  ...
AttributeError: 'Movie' object has no attribute 'author_id'

Maybe with custom Prefetch object?

>>> TaggedItem.objects.all().prefetch_related(
...     Prefetch('content_object', queryset=Book.objects.all().select_related('author')),
...     Prefetch('content_object', queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
  ...
ValueError: Custom queryset can't be used for this lookup.

Some solutions of this problem are shown here. But that’s a lot of massage over the data which I want to avoid.
I really like the API coming from the reversed generic relations, it would be very nice to be able to do prefetchs like that:

>>> TaggedItem.objects.all().prefetch_related(
...     'books__author',
...     'movies__director',
... )
Traceback (most recent call last):
  ...
AttributeError: 'Book' object has no attribute 'object_id'

Or like that:

>>> TaggedItem.objects.all().prefetch_related(
...     Prefetch('books', queryset=Book.objects.all().select_related('author')),
...     Prefetch('movies', queryset=Movie.objects.all().select_related('director')),
... )
Traceback (most recent call last):
  ...
AttributeError: 'Book' object has no attribute 'object_id'

But as you can see, we aways get that AttributeError.
I’m using Django 1.7.3 and Python 2.7.6. And i’m curious why Django is throwing that error? Why is Django searching for an object_id in the Book model?
Why I think this may be a bug?
Usually when we ask prefetch_related to resolve something it can’t, we see:

>>> TaggedItem.objects.all().prefetch_related('some_field')
Traceback (most recent call last):
  ...
AttributeError: Cannot find 'some_field' on TaggedItem object, 'some_field' is an invalid parameter to prefetch_related()

But here, it is different. Django actually tries to resolve the relation… and fails. Is this a bug which should be reported? I have never reported anything to Django so that’s why I’m asking here first. I’m unable to trace the error and decide for myself if this is a bug, or a feature which could be implemented.