Python typing how to apply ABC or Protocol to multiple mixins
Question:
I am trying to extend a certain class, called a ModelSerializer
from Django Rest Framework, but the exact class in not important.
The following example code is to illustrates what I am trying to do. The actual code is too large for it to be meaningful here.
I am adding lots of functionlity so I broke the extra functionality down into 3 mixins:
ConstraintsMixin
– It uses a Constraints
dataclass
FooMixin
AnotherMixin
Then I created an ABC
with the methods/attributes that derived classes must implement called:
ThingsToImplmement
The reason I use an ABC
and not a protocol, is from my research I wish to show errors/linting at the implementation of MyBaseSerializer
(explained below), whereas a protocol would only lint errors at consumption of MyBaseSerializer
, which would be hidden behind Django Rest Framework
generic views, but am willing to a Protocol
if they are more appropriate.
I then tied all the mixins together with the ThingsToImplement
abstract class, along with the rest_framework.ModelSerializer
class into a base class called:
MyBaseSerializer
i.e.:
class MyBaseSerializer(ConstrainsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer):
Now I am trying to add type hints so when a class/mixin refers to attributes/method of another mixin/class/ModelSerializer, it does not show linting errors. Below is with a protocol MySerializerProtocol
which I feel should not be needed, because it pretty much duplicates ThingsToImplement
. The linting errors I get with pyright
are shown as comments. How would one arrange the following code in terms of ABC’s or Protocols to pass linting, and reveal implementation errors when one implements the AbstractSerializer
class? (not some code behind the scenes from Django Rest Framework
will instantiate the implemented AbstractSerializer
).
from abc import ABC, abstractmethod
from typing import Union, Protocol
from dataclasses import dataclass
from rest_framework.serializers import ModelSerializer
@dataclass
class Constraints:
width: int
height: int
class ThingsToImplement(ABC):
my_number: int
constraints: Constraints
@abstractmethod
def get_da_name(self, s:str) -> str:
pass
class MySerializerProtocol(Protocol):
constraints: Constraints
my_number: int
def get_da_name(self) -> str:
...
class ConstraintsMixin:
def __init_subclass__(cls, **kwargs):
if not hasattr(cls, 'constraints'):
raise NotImplementedError('Please add a constraints attribute')
@classmethod
def print_constraints(cls:MySerializerProtocol) -> None:
print(cls.constraints.width, cls.constraints.height)
class FooMixin:
def get_foo(self:MySerializerProtocol) -> str:
return self.get_da_name() if self.constraints.width > 123 else 'Too small to be named'
def get_bar(self: Union[MySerializerProtocol, ModelSerializer]) -> str: # Type of parameter "self" must be a supertype of its class "FooMixin"
# self.to_representation and self.instance are from Model Serializer
return self.to_representation(self.instance) # Cannot access member "to_representation" for type "MySerializerProtocol". Member "to_representation" is unknown
def from_another(self):
return f"from {self.get_number()}" # Cannot access member "get_number" for type "FooMixin". Member "get_number" is unknown.
class AnotherMixin:
def get_number(self:MySerializerProtocol) -> int:
return self.my_number
def from_foo(self):
return "from " + self.get_foo() # Cannot access member "get_foo" for type "AnotherMixin". Member "get_foo" is unknown.
class AbstractSerializer(ConstraintsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer): # The metaclass of a derived class must be a subclass of the metaclasses of all its base classes
def implemented_method_1(self):
return 1
def implemented_method_2(self):
return 2
# ...
def implemented_method_30(self):
return 30
class MySerializer(AbstractSerializer):
my_number = 7
constraints = Constraints(1, 2)
def get_da_name(self, s):
self.number += 1 # Cannot access member "number" for type "MySerializer". Member "number" is unknown.
return 'hi ' + s
Answers:
There are a few issues with the code you showed. I tried to go through those that I thought were most pressing in no particular order.
Avoid nested ABCs if possible
Since AbstractSerializer
will be the abstract base class for your custom serializers, I would suggest defining the abstract methods like get_da_name
on that class directly instead of having them in another, separate ABC like ThingsToImplement
.
It makes the intent clearer because users of that AbstractSerializer
will look at it and immediately see the work they will have to do.
The attributes that need to be present on every serializer subclass like constraints
don’t technically need to be declared on the ABC, but I think it makes sense for the same reason.
The purpose of Protocol
s
I would argue that the one of the main purposes of Protocol
is to simplify doing exactly the things you are doing here. You define common behavior in a protocol that static type checkers can assume is available on a variable annotated with that Protocol
.
In your specific case, it is up to you how finely grained your Protocol
subclasses should be. If you want to be very pedantic, any Mix-in can have its own corresponding Protocol
, but I would argue that is overkill most of the time. It really depends on how complex that "common behavior" becomes, which the Protocol
is supposed to encapsulate.
In your example code I would only define one Protocol
. (see below)
In addition, Protocol
can be used in a generic way, which IMHO fits perfectly into the model serializer context since every serializer will have his instance
set as can be seen in the type stubs for ModelSerializer
(inheriting from BaseSerializer
), which is also generic over a Model
-bound type variable.
Allow ABCs to inherit from ConstraintsMixin
Since you set up your __init_subclass__
class method on ConstraintsMixin
so strictly, you need to ensure that the actual ABC you want to create (i.e. AbstractSerializer
) can inherit from it without triggering the error.
For this you simply add the ABCMeta
check to __init_subclass__
first and avoid triggering the error on ABCs.
Use MySerializerProtocol
in Mix-ins
Since your Mix-ins assume certain behavior in their instance methods, that is exactly where you can use MySerializerProtocol
to annotate the self
parameter.
Again, you may consider splitting the Protocol
up further, if it gets too complex.
Solve the Metaclass conflict
Luckily, this is very easy in this case, since there are only two non-type
Metaclasses involved here, namely the SerializerMetaclass
from Django REST Framework and the ABCMeta
from abc
, and they don’t actually conflict as far as I can see. You just need to define your own Metaclass that inherits from both and specify it in your serializer ABC.
Specify Django Model in subclasses
If you go the generic route (which seems more consistent to me), you should specify the concrete Django Model handled by the serializer, when you subclass AbstractSerializer
.
If you don’t want to go that route, mypy
will complain in --strict
mode upon subclassing ModelSerializer
(that it is missing a type argument), but you can silence that. Also, you can omit the [M]
everywhere in the code (see below) and instead just declare instance: Model
on MySerializerProtocol
.
Fully annotated example code
from abc import ABC, ABCMeta, abstractmethod
from dataclasses import dataclass
from typing import Any, Protocol, TypeVar
from django.db.models import Model
from rest_framework.serializers import ModelSerializer, SerializerMetaclass
M = TypeVar("M", bound=Model)
# Placeholder for a model to be imported from another module:
class ConcreteDjangoModel(Model):
pass
@dataclass
class Constraints:
width: int
height: int
class MySerializerProtocol(Protocol[M]):
"""For type annotations only; generic over `M` like `ModelSerializer`"""
my_number: int
constraints: Constraints
# From ModelSerializer:
instance: M
# From AbstractSerializer:
def get_da_name(self, s: str) -> str: ...
# From FooMixin:
def get_foo(self) -> str: ...
# From AnotherMixin:
def get_number(self) -> int: ...
# From ModelSerializer:
def to_representation(self, instance: M) -> Any: ...
class ConstraintsMixin:
# Class attributes that must be set by subclasses:
constraints: Constraints
def __init_subclass__(cls, **kwargs: Any) -> None:
if not isinstance(cls, ABCMeta) and not hasattr(cls, "constraints"):
raise NotImplementedError("Please add a constraints attribute")
super().__init_subclass__(**kwargs)
@classmethod
def print_constraints(cls: type[MySerializerProtocol[M]]) -> None:
print(cls.constraints.width, cls.constraints.height)
class FooMixin:
def get_foo(self: MySerializerProtocol[M]) -> str:
s = "something"
return self.get_da_name(s) if self.constraints.width > 123 else "Too small to be named"
def get_bar(self: MySerializerProtocol[M]) -> Any:
return self.to_representation(self.instance)
def from_another(self: MySerializerProtocol[M]) -> str:
return f"from {self.get_number()}"
class AnotherMixin:
def get_number(self: MySerializerProtocol[M]) -> int:
return self.my_number
def from_foo(self: MySerializerProtocol[M]) -> str:
return f"from {self.get_foo()}"
class AbstractSerializerMeta(SerializerMetaclass, ABCMeta):
"""To avoid metaclass conflicts in `AbstractSerializer`"""
pass
class AbstractSerializer(
ABC,
ConstraintsMixin,
FooMixin,
AnotherMixin,
ModelSerializer[M],
metaclass=AbstractSerializerMeta,
):
# Class attributes that must be set by subclasses:
constraints: Constraints
@abstractmethod
def get_da_name(self, s: str) -> str: ...
class MySerializer(AbstractSerializer[ConcreteDjangoModel]):
my_number: int = 7
constraints: Constraints = Constraints(1, 2)
def get_da_name(self, s: str) -> str:
self.my_number += 1
return f"hi {s}"
If you have an older Python version (below 3.9
I think), you may need to replace type[MySerializerProtocol[M]]
with typing.Type[MySerializerProtocol[M]]
in the print_constraints
method.
Thanks for the fun little exercise. Hope this helps.
Feel free to comment, if something is unclear. I will try to amend my answer if necessary.
I am trying to extend a certain class, called a ModelSerializer
from Django Rest Framework, but the exact class in not important.
The following example code is to illustrates what I am trying to do. The actual code is too large for it to be meaningful here.
I am adding lots of functionlity so I broke the extra functionality down into 3 mixins:
ConstraintsMixin
– It uses aConstraints
dataclassFooMixin
AnotherMixin
Then I created an ABC
with the methods/attributes that derived classes must implement called:
ThingsToImplmement
The reason I use an ABC
and not a protocol, is from my research I wish to show errors/linting at the implementation of MyBaseSerializer
(explained below), whereas a protocol would only lint errors at consumption of MyBaseSerializer
, which would be hidden behind Django Rest Framework
generic views, but am willing to a Protocol
if they are more appropriate.
I then tied all the mixins together with the ThingsToImplement
abstract class, along with the rest_framework.ModelSerializer
class into a base class called:
MyBaseSerializer
i.e.:
class MyBaseSerializer(ConstrainsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer):
Now I am trying to add type hints so when a class/mixin refers to attributes/method of another mixin/class/ModelSerializer, it does not show linting errors. Below is with a protocol MySerializerProtocol
which I feel should not be needed, because it pretty much duplicates ThingsToImplement
. The linting errors I get with pyright
are shown as comments. How would one arrange the following code in terms of ABC’s or Protocols to pass linting, and reveal implementation errors when one implements the AbstractSerializer
class? (not some code behind the scenes from Django Rest Framework
will instantiate the implemented AbstractSerializer
).
from abc import ABC, abstractmethod
from typing import Union, Protocol
from dataclasses import dataclass
from rest_framework.serializers import ModelSerializer
@dataclass
class Constraints:
width: int
height: int
class ThingsToImplement(ABC):
my_number: int
constraints: Constraints
@abstractmethod
def get_da_name(self, s:str) -> str:
pass
class MySerializerProtocol(Protocol):
constraints: Constraints
my_number: int
def get_da_name(self) -> str:
...
class ConstraintsMixin:
def __init_subclass__(cls, **kwargs):
if not hasattr(cls, 'constraints'):
raise NotImplementedError('Please add a constraints attribute')
@classmethod
def print_constraints(cls:MySerializerProtocol) -> None:
print(cls.constraints.width, cls.constraints.height)
class FooMixin:
def get_foo(self:MySerializerProtocol) -> str:
return self.get_da_name() if self.constraints.width > 123 else 'Too small to be named'
def get_bar(self: Union[MySerializerProtocol, ModelSerializer]) -> str: # Type of parameter "self" must be a supertype of its class "FooMixin"
# self.to_representation and self.instance are from Model Serializer
return self.to_representation(self.instance) # Cannot access member "to_representation" for type "MySerializerProtocol". Member "to_representation" is unknown
def from_another(self):
return f"from {self.get_number()}" # Cannot access member "get_number" for type "FooMixin". Member "get_number" is unknown.
class AnotherMixin:
def get_number(self:MySerializerProtocol) -> int:
return self.my_number
def from_foo(self):
return "from " + self.get_foo() # Cannot access member "get_foo" for type "AnotherMixin". Member "get_foo" is unknown.
class AbstractSerializer(ConstraintsMixin, FooMixin, AnotherMixin, ThingsToImplement, ModelSerializer): # The metaclass of a derived class must be a subclass of the metaclasses of all its base classes
def implemented_method_1(self):
return 1
def implemented_method_2(self):
return 2
# ...
def implemented_method_30(self):
return 30
class MySerializer(AbstractSerializer):
my_number = 7
constraints = Constraints(1, 2)
def get_da_name(self, s):
self.number += 1 # Cannot access member "number" for type "MySerializer". Member "number" is unknown.
return 'hi ' + s
There are a few issues with the code you showed. I tried to go through those that I thought were most pressing in no particular order.
Avoid nested ABCs if possible
Since AbstractSerializer
will be the abstract base class for your custom serializers, I would suggest defining the abstract methods like get_da_name
on that class directly instead of having them in another, separate ABC like ThingsToImplement
.
It makes the intent clearer because users of that AbstractSerializer
will look at it and immediately see the work they will have to do.
The attributes that need to be present on every serializer subclass like constraints
don’t technically need to be declared on the ABC, but I think it makes sense for the same reason.
The purpose of Protocol
s
I would argue that the one of the main purposes of Protocol
is to simplify doing exactly the things you are doing here. You define common behavior in a protocol that static type checkers can assume is available on a variable annotated with that Protocol
.
In your specific case, it is up to you how finely grained your Protocol
subclasses should be. If you want to be very pedantic, any Mix-in can have its own corresponding Protocol
, but I would argue that is overkill most of the time. It really depends on how complex that "common behavior" becomes, which the Protocol
is supposed to encapsulate.
In your example code I would only define one Protocol
. (see below)
In addition, Protocol
can be used in a generic way, which IMHO fits perfectly into the model serializer context since every serializer will have his instance
set as can be seen in the type stubs for ModelSerializer
(inheriting from BaseSerializer
), which is also generic over a Model
-bound type variable.
Allow ABCs to inherit from ConstraintsMixin
Since you set up your __init_subclass__
class method on ConstraintsMixin
so strictly, you need to ensure that the actual ABC you want to create (i.e. AbstractSerializer
) can inherit from it without triggering the error.
For this you simply add the ABCMeta
check to __init_subclass__
first and avoid triggering the error on ABCs.
Use MySerializerProtocol
in Mix-ins
Since your Mix-ins assume certain behavior in their instance methods, that is exactly where you can use MySerializerProtocol
to annotate the self
parameter.
Again, you may consider splitting the Protocol
up further, if it gets too complex.
Solve the Metaclass conflict
Luckily, this is very easy in this case, since there are only two non-type
Metaclasses involved here, namely the SerializerMetaclass
from Django REST Framework and the ABCMeta
from abc
, and they don’t actually conflict as far as I can see. You just need to define your own Metaclass that inherits from both and specify it in your serializer ABC.
Specify Django Model in subclasses
If you go the generic route (which seems more consistent to me), you should specify the concrete Django Model handled by the serializer, when you subclass AbstractSerializer
.
If you don’t want to go that route, mypy
will complain in --strict
mode upon subclassing ModelSerializer
(that it is missing a type argument), but you can silence that. Also, you can omit the [M]
everywhere in the code (see below) and instead just declare instance: Model
on MySerializerProtocol
.
Fully annotated example code
from abc import ABC, ABCMeta, abstractmethod
from dataclasses import dataclass
from typing import Any, Protocol, TypeVar
from django.db.models import Model
from rest_framework.serializers import ModelSerializer, SerializerMetaclass
M = TypeVar("M", bound=Model)
# Placeholder for a model to be imported from another module:
class ConcreteDjangoModel(Model):
pass
@dataclass
class Constraints:
width: int
height: int
class MySerializerProtocol(Protocol[M]):
"""For type annotations only; generic over `M` like `ModelSerializer`"""
my_number: int
constraints: Constraints
# From ModelSerializer:
instance: M
# From AbstractSerializer:
def get_da_name(self, s: str) -> str: ...
# From FooMixin:
def get_foo(self) -> str: ...
# From AnotherMixin:
def get_number(self) -> int: ...
# From ModelSerializer:
def to_representation(self, instance: M) -> Any: ...
class ConstraintsMixin:
# Class attributes that must be set by subclasses:
constraints: Constraints
def __init_subclass__(cls, **kwargs: Any) -> None:
if not isinstance(cls, ABCMeta) and not hasattr(cls, "constraints"):
raise NotImplementedError("Please add a constraints attribute")
super().__init_subclass__(**kwargs)
@classmethod
def print_constraints(cls: type[MySerializerProtocol[M]]) -> None:
print(cls.constraints.width, cls.constraints.height)
class FooMixin:
def get_foo(self: MySerializerProtocol[M]) -> str:
s = "something"
return self.get_da_name(s) if self.constraints.width > 123 else "Too small to be named"
def get_bar(self: MySerializerProtocol[M]) -> Any:
return self.to_representation(self.instance)
def from_another(self: MySerializerProtocol[M]) -> str:
return f"from {self.get_number()}"
class AnotherMixin:
def get_number(self: MySerializerProtocol[M]) -> int:
return self.my_number
def from_foo(self: MySerializerProtocol[M]) -> str:
return f"from {self.get_foo()}"
class AbstractSerializerMeta(SerializerMetaclass, ABCMeta):
"""To avoid metaclass conflicts in `AbstractSerializer`"""
pass
class AbstractSerializer(
ABC,
ConstraintsMixin,
FooMixin,
AnotherMixin,
ModelSerializer[M],
metaclass=AbstractSerializerMeta,
):
# Class attributes that must be set by subclasses:
constraints: Constraints
@abstractmethod
def get_da_name(self, s: str) -> str: ...
class MySerializer(AbstractSerializer[ConcreteDjangoModel]):
my_number: int = 7
constraints: Constraints = Constraints(1, 2)
def get_da_name(self, s: str) -> str:
self.my_number += 1
return f"hi {s}"
If you have an older Python version (below 3.9
I think), you may need to replace type[MySerializerProtocol[M]]
with typing.Type[MySerializerProtocol[M]]
in the print_constraints
method.
Thanks for the fun little exercise. Hope this helps.
Feel free to comment, if something is unclear. I will try to amend my answer if necessary.