Python typing how to apply ABC or Protocol to multiple mixins

Question:

I am trying to extend a certain class, called a ModelSerializer from Django Rest Framework, but the exact class in not important.

The following example code is to illustrates what I am trying to do. The actual code is too large for it to be meaningful here.

I am adding lots of functionlity so I broke the extra functionality down into 3 mixins:

  • ConstraintsMixin – It uses a Constraints dataclass
  • FooMixin
  • AnotherMixin

Then I created an ABC with the methods/attributes that derived classes must implement called:

  • ThingsToImplmement

The reason I use an ABC and not a protocol, is from my research I wish to show errors/linting at the implementation of MyBaseSerializer (explained below), whereas a protocol would only lint errors at consumption of MyBaseSerializer, which would be hidden behind Django Rest Framework generic views, but am willing to a Protocol if they are more appropriate.

I then tied all the mixins together with the ThingsToImplement abstract class, along with the rest_framework.ModelSerializer class into a base class called:

  • MyBaseSerializer

i.e.:

class MyBaseSerializer(ConstrainsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer):

Now I am trying to add type hints so when a class/mixin refers to attributes/method of another mixin/class/ModelSerializer, it does not show linting errors. Below is with a protocol MySerializerProtocol which I feel should not be needed, because it pretty much duplicates ThingsToImplement. The linting errors I get with pyright are shown as comments. How would one arrange the following code in terms of ABC’s or Protocols to pass linting, and reveal implementation errors when one implements the AbstractSerializer class? (not some code behind the scenes from Django Rest Framework will instantiate the implemented AbstractSerializer).

from abc import ABC, abstractmethod
from typing import Union, Protocol
from dataclasses import dataclass
from rest_framework.serializers import ModelSerializer


@dataclass
class Constraints:
    width: int
    height: int

class ThingsToImplement(ABC):
    my_number: int
    constraints: Constraints

    @abstractmethod
    def get_da_name(self, s:str) -> str:
        pass

class MySerializerProtocol(Protocol):
    constraints: Constraints
    my_number: int

    def get_da_name(self) -> str:
        ...

class ConstraintsMixin:
    def __init_subclass__(cls, **kwargs):
        if not hasattr(cls, 'constraints'):
            raise NotImplementedError('Please add a constraints attribute')

    @classmethod
    def print_constraints(cls:MySerializerProtocol) -> None:
        print(cls.constraints.width, cls.constraints.height)


class FooMixin:
    def get_foo(self:MySerializerProtocol) -> str:
        return self.get_da_name() if self.constraints.width > 123 else 'Too small to be named'

    def get_bar(self: Union[MySerializerProtocol, ModelSerializer]) -> str:  # Type of parameter "self" must be a supertype of its class "FooMixin"
        # self.to_representation and self.instance are from Model Serializer
        return self.to_representation(self.instance)  # Cannot access member "to_representation" for type "MySerializerProtocol". Member "to_representation" is unknown

    def from_another(self):
        return f"from {self.get_number()}"  # Cannot access member "get_number" for type "FooMixin". Member "get_number" is unknown.

class AnotherMixin:
    def get_number(self:MySerializerProtocol) -> int:
        return self.my_number

    def from_foo(self):
        return "from " + self.get_foo()  # Cannot access member "get_foo" for type "AnotherMixin". Member "get_foo" is unknown.


class AbstractSerializer(ConstraintsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer): # The metaclass of a derived class must be a subclass of the metaclasses of all its base classes
    def implemented_method_1(self):
        return 1

    def implemented_method_2(self):
        return 2

    # ...
    def implemented_method_30(self):
        return 30


class MySerializer(AbstractSerializer):
    my_number = 7
    constraints = Constraints(1, 2)

    def get_da_name(self, s):
        self.number += 1  # Cannot access member "number" for type "MySerializer". Member "number" is unknown.
        return 'hi ' + s
Asked By: run_the_race

||

Answers:

There are a few issues with the code you showed. I tried to go through those that I thought were most pressing in no particular order.

Avoid nested ABCs if possible

Since AbstractSerializer will be the abstract base class for your custom serializers, I would suggest defining the abstract methods like get_da_name on that class directly instead of having them in another, separate ABC like ThingsToImplement.

It makes the intent clearer because users of that AbstractSerializer will look at it and immediately see the work they will have to do.

The attributes that need to be present on every serializer subclass like constraints don’t technically need to be declared on the ABC, but I think it makes sense for the same reason.

The purpose of Protocols

I would argue that the one of the main purposes of Protocol is to simplify doing exactly the things you are doing here. You define common behavior in a protocol that static type checkers can assume is available on a variable annotated with that Protocol.

In your specific case, it is up to you how finely grained your Protocol subclasses should be. If you want to be very pedantic, any Mix-in can have its own corresponding Protocol, but I would argue that is overkill most of the time. It really depends on how complex that "common behavior" becomes, which the Protocol is supposed to encapsulate.

In your example code I would only define one Protocol. (see below)

In addition, Protocol can be used in a generic way, which IMHO fits perfectly into the model serializer context since every serializer will have his instance set as can be seen in the type stubs for ModelSerializer (inheriting from BaseSerializer), which is also generic over a Model-bound type variable.

Allow ABCs to inherit from ConstraintsMixin

Since you set up your __init_subclass__ class method on ConstraintsMixin so strictly, you need to ensure that the actual ABC you want to create (i.e. AbstractSerializer) can inherit from it without triggering the error.

For this you simply add the ABCMeta check to __init_subclass__ first and avoid triggering the error on ABCs.

Use MySerializerProtocol in Mix-ins

Since your Mix-ins assume certain behavior in their instance methods, that is exactly where you can use MySerializerProtocol to annotate the self parameter.

Again, you may consider splitting the Protocol up further, if it gets too complex.

Solve the Metaclass conflict

Luckily, this is very easy in this case, since there are only two non-type Metaclasses involved here, namely the SerializerMetaclass from Django REST Framework and the ABCMeta from abc, and they don’t actually conflict as far as I can see. You just need to define your own Metaclass that inherits from both and specify it in your serializer ABC.

Specify Django Model in subclasses

If you go the generic route (which seems more consistent to me), you should specify the concrete Django Model handled by the serializer, when you subclass AbstractSerializer.

If you don’t want to go that route, mypy will complain in --strict mode upon subclassing ModelSerializer (that it is missing a type argument), but you can silence that. Also, you can omit the [M] everywhere in the code (see below) and instead just declare instance: Model on MySerializerProtocol.

Fully annotated example code

from abc import ABC, ABCMeta, abstractmethod
from dataclasses import dataclass
from typing import Any, Protocol, TypeVar

from django.db.models import Model
from rest_framework.serializers import ModelSerializer, SerializerMetaclass


M = TypeVar("M", bound=Model)


# Placeholder for a model to be imported from another module:
class ConcreteDjangoModel(Model):
    pass


@dataclass
class Constraints:
    width: int
    height: int


class MySerializerProtocol(Protocol[M]):
    """For type annotations only; generic over `M` like `ModelSerializer`"""
    my_number: int
    constraints: Constraints
    # From ModelSerializer:
    instance: M

    # From AbstractSerializer:
    def get_da_name(self, s: str) -> str: ...
    # From FooMixin:
    def get_foo(self) -> str: ...
    # From AnotherMixin:
    def get_number(self) -> int: ...
    # From ModelSerializer:
    def to_representation(self, instance: M) -> Any: ...


class ConstraintsMixin:
    # Class attributes that must be set by subclasses:
    constraints: Constraints

    def __init_subclass__(cls, **kwargs: Any) -> None:
        if not isinstance(cls, ABCMeta) and not hasattr(cls, "constraints"):
            raise NotImplementedError("Please add a constraints attribute")
        super().__init_subclass__(**kwargs)

    @classmethod
    def print_constraints(cls: type[MySerializerProtocol[M]]) -> None:
        print(cls.constraints.width, cls.constraints.height)


class FooMixin:
    def get_foo(self: MySerializerProtocol[M]) -> str:
        s = "something"
        return self.get_da_name(s) if self.constraints.width > 123 else "Too small to be named"

    def get_bar(self: MySerializerProtocol[M]) -> Any:
        return self.to_representation(self.instance)

    def from_another(self: MySerializerProtocol[M]) -> str:
        return f"from {self.get_number()}"


class AnotherMixin:
    def get_number(self: MySerializerProtocol[M]) -> int:
        return self.my_number

    def from_foo(self: MySerializerProtocol[M]) -> str:
        return f"from {self.get_foo()}"


class AbstractSerializerMeta(SerializerMetaclass, ABCMeta):
    """To avoid metaclass conflicts in `AbstractSerializer`"""
    pass


class AbstractSerializer(
    ABC,
    ConstraintsMixin,
    FooMixin,
    AnotherMixin,
    ModelSerializer[M],
    metaclass=AbstractSerializerMeta,
):
    # Class attributes that must be set by subclasses:
    constraints: Constraints

    @abstractmethod
    def get_da_name(self, s: str) -> str: ...


class MySerializer(AbstractSerializer[ConcreteDjangoModel]):
    my_number: int = 7
    constraints: Constraints = Constraints(1, 2)

    def get_da_name(self, s: str) -> str:
        self.my_number += 1
        return f"hi {s}"

If you have an older Python version (below 3.9 I think), you may need to replace type[MySerializerProtocol[M]] with typing.Type[MySerializerProtocol[M]] in the print_constraints method.


Thanks for the fun little exercise. Hope this helps.

Feel free to comment, if something is unclear. I will try to amend my answer if necessary.

Answered By: Daniil Fajnberg
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.