How to make mypy like my protocols that work at runtime with runtime_checkable
Question:
I defined a couple of protocols like so:
import json
from typing import Any, Protocol, TypeVar, runtime_checkable
T_co = TypeVar('T_co', covariant=True)
T = TypeVar('T')
@runtime_checkable
class SupportsRead(Protocol[T_co]):
def read(self, __length: int = ...) -> T_co: ...
@runtime_checkable
class SupportsWrite(Protocol[T_co]):
def write(self, data: str | bytes): ...
@runtime_checkable
class SerializerToString(Protocol):
def dumps(self, value: Any, *argv, **kwargs) -> str: ...
def loads(self, value: str | bytes, *argv, **kwargs) -> Any: ...
@runtime_checkable
class SerializerToFile(Protocol):
def dump(self, value: Any, file: SupportsWrite[str | bytes], **kwargs) -> None: ...
def load(self, file: SupportsRead[str | bytes], **kwargs) -> Any: ...
@runtime_checkable
class Serializer(SerializerToString, SerializerToFile, Protocol):
pass
class MySerializer:
def dumps(self, value: Any) -> str:
return f"dumps {value}"
def loads(self, value: str) -> Any:
return f"loads {value}"
var1: SerializerToFile = json
var2: SerializerToString = json
var3: Serializer = json
var4: SerializerToString = MySerializer()
assert isinstance(var1, SerializerToFile)
assert isinstance(var2, SerializerToString)
assert isinstance(var3, Serializer)
assert isinstance(var4, SerializerToString)
print("everything ok")
It runs without error, but mypy marks all the assigments to var1–var4 with errors like so:
test11.py:42: error: Incompatible types in assignment (expression has type Module, variable has type "SerializerToFile") [assignment]
test11.py:42: note: Following member(s) of "Module json" have conflicts:
test11.py:42: note: Expected:
test11.py:42: note: def dump(value: Any, file: SupportsWrite[Union[str, bytes]], **kwargs: Any) -> None
test11.py:42: note: Got:
test11.py:42: note: def dump(obj: Any, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> None
test11.py:42: note: Expected:
test11.py:42: note: def load(file: SupportsRead[Union[str, bytes]], **kwargs: Any) -> Any
test11.py:42: note: Got:
test11.py:42: note: def load(fp: SupportsRead[Union[str, bytes]], *, cls: Optional[Type[JSONDecoder]] = ..., object_hook: Optional[Callable[[Dict[Any, Any]], Any]] = ..., parse_float: Optional[Callable[[str], Any]] = ..., parse_int: Optional[Callable[[str], Any]] = ..., parse_constant: Optional[Callable[[str], Any]] = ..., object_pairs_hook: Optional[Callable[[List[Tuple[Any, Any]]], Any]] = ..., **kwds: Any) -> Any
test11.py:43: error: Incompatible types in assignment (expression has type Module, variable has type "SerializerToString") [assignment]
test11.py:43: note: Following member(s) of "Module json" have conflicts:
test11.py:43: note: Expected:
test11.py:43: note: def dumps(value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:43: note: Got:
test11.py:43: note: def dumps(obj: Any, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
test11.py:43: note: Expected:
test11.py:43: note: def loads(value: Union[str, bytes], *argv: Any, **kwargs: Any) -> Any
test11.py:43: note: Got:
test11.py:43: note: def loads(s: Union[str, bytes, bytearray], *, cls: Optional[Type[JSONDecoder]] = ..., object_hook: Optional[Callable[[Dict[Any, Any]], Any]] = ..., parse_float: Optional[Callable[[str], Any]] = ..., parse_int: Optional[Callable[[str], Any]] = ..., parse_constant: Optional[Callable[[str], Any]] = ..., object_pairs_hook: Optional[Callable[[List[Tuple[Any, Any]]], Any]] = ..., **kwds: Any) -> Any
test11.py:44: error: Incompatible types in assignment (expression has type Module, variable has type "Serializer") [assignment]
test11.py:44: note: Following member(s) of "Module json" have conflicts:
test11.py:44: note: Expected:
test11.py:44: note: def dump(value: Any, file: SupportsWrite[Union[str, bytes]], **kwargs: Any) -> None
test11.py:44: note: Got:
test11.py:44: note: def dump(obj: Any, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> None
test11.py:44: note: Expected:
test11.py:44: note: def dumps(value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:44: note: Got:
test11.py:44: note: def dumps(obj: Any, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
test11.py:44: note: <2 more conflict(s) not shown>
test11.py:45: error: Incompatible types in assignment (expression has type "MySerializer", variable has type "SerializerToString") [assignment]
test11.py:45: note: Following member(s) of "MySerializer" have conflicts:
test11.py:45: note: Expected:
test11.py:45: note: def dumps(self, value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:45: note: Got:
test11.py:45: note: def dumps(self, value: Any) -> str
test11.py:45: note: Expected:
test11.py:45: note: def loads(self, value: Union[str, bytes], *argv: Any, **kwargs: Any) -> Any
test11.py:45: note: Got:
test11.py:45: note: def loads(self, value: str) -> Any
Found 4 errors in 1 file (checked 1 source file)
Is there a way to please mypy with those protocols?
Answers:
I’ll go through the problems one by one.
Given the following protocols for file-like objects:
from typing import Any, Protocol, TypeVar, runtime_checkable
T_co = TypeVar('T_co', covariant=True)
@runtime_checkable
class SupportsRead(Protocol[T_co]):
def read(self, __length: int = ...) -> T_co: ...
@runtime_checkable
class SupportsWrite(Protocol[T_co]):
def write(self, data: str | bytes) -> Any: ...
SerializerToFile
Simply adjusting your protocol members’ parameter names to reflect those of the corresponding functions in the json module fixes the issue:
@runtime_checkable
class SerializerToFile(Protocol):
def dump(
self,
obj: Any, # was called `value`
fp: SupportsWrite[str | bytes], # was called `file`
**kwargs: Any, # was missing `Any` (optional)
) -> None: ...
def load(
self,
fp: SupportsRead[str | bytes], # was called `file`
**kwargs: Any, # was missing `Any` (optional)
) -> Any: ...
import json
var1: SerializerToFile = json # passes
SerializerToString
Again, the names did not match, but also your protocol methods accept arbitrary positional arguments, which neither the loads nor the dumps function does:
@runtime_checkable
class SerializerToString(Protocol):
def dumps(
self,
obj: Any, # was called `value`
# *argv: Any, # this needs to be removed
**kwargs: Any, # was missing `Any` (optional)
) -> str: ...
def loads(
self,
s: str | bytes, # was called `value`
# *argv: Any, # this needs to be removed
**kwargs: Any, # was missing `Any` (optional)
) -> Any: ...
import json
var1: SerializerToString = json # passes
Serializer (intersection)
With the fixed protocols from above, the intersection of both of them is also a valid annotation to use for the json module:
...
@runtime_checkable
class Serializer(SerializerToString, SerializerToFile, Protocol):
pass
import json
var1: Serializer = json # passes
Custom serializer
Now the custom serializer needs to be adjusted accordingly, so that it matches the protocol as well:
...
class MySerializer:
def dumps(self, obj: Any, **_kwargs: Any) -> str:
return f"dumps {obj}"
def loads(self, s: str | bytes, **_kwargs: Any) -> Any:
return f"loads {s!r}"
var1: SerializerToString = MySerializer() # passes
The first loads argument must accept bytes as well, since the protocol defines it as str | bytes. Subtypes are contravariant in their method parameters. So you could only widen the parameter types accepted by the serializer, never narrow it. Something silly like str | bytes | tuple[Any] for the first argument on your loads method would still work, but just str does not.
Attempt at explanation
As opposed to the Callable type, where you may only define the types that parameters in certain positions of the callable take, the protocol methods are more specific, since Python does allow all "normal" arguments to a function call to be passed as keyword-arguments (i.e. by name).
The exceptions are variable-number positional arguments, defined via *args and arguments specifically defined as positional-only by having their parameters precede a / in the signature.
So if you say that something is a structural subtype of e.g. SerializerToFile, if it implements a method dumps, where the first parameter is "positional-or-keyword" and named value, you could then use an object of that type somewhere and call that method as dumps(value=123). But the json.dumps method can not be called that way because its first parameter is named obj, so a call of json.dumps(obj={}) works, whereas the call json.dumps(value={}) would fail. Therefore json does not conform to the protocol.
Similarly, if your protocol allows "var. args" in a method, i.e. dumps(obj: Any, *args: Any, **kwargs: Any), that means you might want to call that method on an instance of that type as dumps({}, 1, 2, 3, 4), but that is not allowed for json.dumps because aside from the very first parameter it explicitly defines the rest as keyword-only (all following the *).
I defined a couple of protocols like so:
import json
from typing import Any, Protocol, TypeVar, runtime_checkable
T_co = TypeVar('T_co', covariant=True)
T = TypeVar('T')
@runtime_checkable
class SupportsRead(Protocol[T_co]):
def read(self, __length: int = ...) -> T_co: ...
@runtime_checkable
class SupportsWrite(Protocol[T_co]):
def write(self, data: str | bytes): ...
@runtime_checkable
class SerializerToString(Protocol):
def dumps(self, value: Any, *argv, **kwargs) -> str: ...
def loads(self, value: str | bytes, *argv, **kwargs) -> Any: ...
@runtime_checkable
class SerializerToFile(Protocol):
def dump(self, value: Any, file: SupportsWrite[str | bytes], **kwargs) -> None: ...
def load(self, file: SupportsRead[str | bytes], **kwargs) -> Any: ...
@runtime_checkable
class Serializer(SerializerToString, SerializerToFile, Protocol):
pass
class MySerializer:
def dumps(self, value: Any) -> str:
return f"dumps {value}"
def loads(self, value: str) -> Any:
return f"loads {value}"
var1: SerializerToFile = json
var2: SerializerToString = json
var3: Serializer = json
var4: SerializerToString = MySerializer()
assert isinstance(var1, SerializerToFile)
assert isinstance(var2, SerializerToString)
assert isinstance(var3, Serializer)
assert isinstance(var4, SerializerToString)
print("everything ok")
It runs without error, but mypy marks all the assigments to var1–var4 with errors like so:
test11.py:42: error: Incompatible types in assignment (expression has type Module, variable has type "SerializerToFile") [assignment]
test11.py:42: note: Following member(s) of "Module json" have conflicts:
test11.py:42: note: Expected:
test11.py:42: note: def dump(value: Any, file: SupportsWrite[Union[str, bytes]], **kwargs: Any) -> None
test11.py:42: note: Got:
test11.py:42: note: def dump(obj: Any, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> None
test11.py:42: note: Expected:
test11.py:42: note: def load(file: SupportsRead[Union[str, bytes]], **kwargs: Any) -> Any
test11.py:42: note: Got:
test11.py:42: note: def load(fp: SupportsRead[Union[str, bytes]], *, cls: Optional[Type[JSONDecoder]] = ..., object_hook: Optional[Callable[[Dict[Any, Any]], Any]] = ..., parse_float: Optional[Callable[[str], Any]] = ..., parse_int: Optional[Callable[[str], Any]] = ..., parse_constant: Optional[Callable[[str], Any]] = ..., object_pairs_hook: Optional[Callable[[List[Tuple[Any, Any]]], Any]] = ..., **kwds: Any) -> Any
test11.py:43: error: Incompatible types in assignment (expression has type Module, variable has type "SerializerToString") [assignment]
test11.py:43: note: Following member(s) of "Module json" have conflicts:
test11.py:43: note: Expected:
test11.py:43: note: def dumps(value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:43: note: Got:
test11.py:43: note: def dumps(obj: Any, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
test11.py:43: note: Expected:
test11.py:43: note: def loads(value: Union[str, bytes], *argv: Any, **kwargs: Any) -> Any
test11.py:43: note: Got:
test11.py:43: note: def loads(s: Union[str, bytes, bytearray], *, cls: Optional[Type[JSONDecoder]] = ..., object_hook: Optional[Callable[[Dict[Any, Any]], Any]] = ..., parse_float: Optional[Callable[[str], Any]] = ..., parse_int: Optional[Callable[[str], Any]] = ..., parse_constant: Optional[Callable[[str], Any]] = ..., object_pairs_hook: Optional[Callable[[List[Tuple[Any, Any]]], Any]] = ..., **kwds: Any) -> Any
test11.py:44: error: Incompatible types in assignment (expression has type Module, variable has type "Serializer") [assignment]
test11.py:44: note: Following member(s) of "Module json" have conflicts:
test11.py:44: note: Expected:
test11.py:44: note: def dump(value: Any, file: SupportsWrite[Union[str, bytes]], **kwargs: Any) -> None
test11.py:44: note: Got:
test11.py:44: note: def dump(obj: Any, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> None
test11.py:44: note: Expected:
test11.py:44: note: def dumps(value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:44: note: Got:
test11.py:44: note: def dumps(obj: Any, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: Optional[Type[JSONEncoder]] = ..., indent: Union[None, int, str] = ..., separators: Optional[Tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
test11.py:44: note: <2 more conflict(s) not shown>
test11.py:45: error: Incompatible types in assignment (expression has type "MySerializer", variable has type "SerializerToString") [assignment]
test11.py:45: note: Following member(s) of "MySerializer" have conflicts:
test11.py:45: note: Expected:
test11.py:45: note: def dumps(self, value: Any, *argv: Any, **kwargs: Any) -> str
test11.py:45: note: Got:
test11.py:45: note: def dumps(self, value: Any) -> str
test11.py:45: note: Expected:
test11.py:45: note: def loads(self, value: Union[str, bytes], *argv: Any, **kwargs: Any) -> Any
test11.py:45: note: Got:
test11.py:45: note: def loads(self, value: str) -> Any
Found 4 errors in 1 file (checked 1 source file)
Is there a way to please mypy with those protocols?
I’ll go through the problems one by one.
Given the following protocols for file-like objects:
from typing import Any, Protocol, TypeVar, runtime_checkable
T_co = TypeVar('T_co', covariant=True)
@runtime_checkable
class SupportsRead(Protocol[T_co]):
def read(self, __length: int = ...) -> T_co: ...
@runtime_checkable
class SupportsWrite(Protocol[T_co]):
def write(self, data: str | bytes) -> Any: ...
SerializerToFile
Simply adjusting your protocol members’ parameter names to reflect those of the corresponding functions in the json module fixes the issue:
@runtime_checkable
class SerializerToFile(Protocol):
def dump(
self,
obj: Any, # was called `value`
fp: SupportsWrite[str | bytes], # was called `file`
**kwargs: Any, # was missing `Any` (optional)
) -> None: ...
def load(
self,
fp: SupportsRead[str | bytes], # was called `file`
**kwargs: Any, # was missing `Any` (optional)
) -> Any: ...
import json
var1: SerializerToFile = json # passes
SerializerToString
Again, the names did not match, but also your protocol methods accept arbitrary positional arguments, which neither the loads nor the dumps function does:
@runtime_checkable
class SerializerToString(Protocol):
def dumps(
self,
obj: Any, # was called `value`
# *argv: Any, # this needs to be removed
**kwargs: Any, # was missing `Any` (optional)
) -> str: ...
def loads(
self,
s: str | bytes, # was called `value`
# *argv: Any, # this needs to be removed
**kwargs: Any, # was missing `Any` (optional)
) -> Any: ...
import json
var1: SerializerToString = json # passes
Serializer (intersection)
With the fixed protocols from above, the intersection of both of them is also a valid annotation to use for the json module:
...
@runtime_checkable
class Serializer(SerializerToString, SerializerToFile, Protocol):
pass
import json
var1: Serializer = json # passes
Custom serializer
Now the custom serializer needs to be adjusted accordingly, so that it matches the protocol as well:
...
class MySerializer:
def dumps(self, obj: Any, **_kwargs: Any) -> str:
return f"dumps {obj}"
def loads(self, s: str | bytes, **_kwargs: Any) -> Any:
return f"loads {s!r}"
var1: SerializerToString = MySerializer() # passes
The first loads argument must accept bytes as well, since the protocol defines it as str | bytes. Subtypes are contravariant in their method parameters. So you could only widen the parameter types accepted by the serializer, never narrow it. Something silly like str | bytes | tuple[Any] for the first argument on your loads method would still work, but just str does not.
Attempt at explanation
As opposed to the Callable type, where you may only define the types that parameters in certain positions of the callable take, the protocol methods are more specific, since Python does allow all "normal" arguments to a function call to be passed as keyword-arguments (i.e. by name).
The exceptions are variable-number positional arguments, defined via *args and arguments specifically defined as positional-only by having their parameters precede a / in the signature.
So if you say that something is a structural subtype of e.g. SerializerToFile, if it implements a method dumps, where the first parameter is "positional-or-keyword" and named value, you could then use an object of that type somewhere and call that method as dumps(value=123). But the json.dumps method can not be called that way because its first parameter is named obj, so a call of json.dumps(obj={}) works, whereas the call json.dumps(value={}) would fail. Therefore json does not conform to the protocol.
Similarly, if your protocol allows "var. args" in a method, i.e. dumps(obj: Any, *args: Any, **kwargs: Any), that means you might want to call that method on an instance of that type as dumps({}, 1, 2, 3, 4), but that is not allowed for json.dumps because aside from the very first parameter it explicitly defines the rest as keyword-only (all following the *).