mypy: argument of method incompatible with supertype
Question:
Look at the example code (mypy_test.py):
import typing
class Base:
def fun(self, a: str):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
now mypy complains:
mypy mypy_test.py
mypy_test.py:10: error: Argument 1 of "fun" incompatible with supertype "Base"
In that case, how can I work with class hierarchy and be type safe?
Software versions:
mypy 0.650
Python 3.7.1
What I tried:
import typing
class Base:
def fun(self, a: typing.Type[str]):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
But it didn’t help.
One user commented: “Looks like you cannot narrow down accepted types in overridden method?”
But in that case, if I use the broadest type possible (typing.Any) in a base class signature it shoudn’t work either. But it does:
import typing
class Base:
def fun(self, a: typing.Any):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
No complains from mypy with the code directly above.
Answers:
Use a generic class as follows:
from typing import Generic
from typing import NewType
from typing import TypeVar
BoundedStr = TypeVar('BoundedStr', bound=str)
class Base(Generic[BoundedStr]):
def fun(self, a: BoundedStr) -> None:
pass
SomeType = NewType('SomeType', str)
class Derived(Base[SomeType]):
def fun(self, a: SomeType) -> None:
pass
The idea is to define a base class, with a generic type. Now you want this generic type to be a subtype of str, hence the bound=str directive.
Then you define your type SomeType, and when you subclass Base, you specify what the generic type variable is: in this case it’s SomeType. Then mypy checks that SomeType is a subtype of str (since we stated that BoundedStr must be a bounded by str), and in this case mypy is happy.
Of course, mypy will complain if you defined SomeType = NewType('SomeType', int) and used it as the type variable for Base or, more generally, if you subclass Base[SomeTypeVariable] if SomeTypeVariable is not a subtype of str.
I read in a comment that you want to ditch mypy. Don’t! Instead, learn how types work; when you’re in situations where you feel mypy is against you, it’s very likely that there’s something you didn’t quite understand. In this case seek some help from other people instead of giving up!
Both functions have the same name, so just rename 1 of the functions. mypy will give you the same error if you do this:
class Derived(Base):
def fun(self, a: int):
Renaming fun to fun1 solves the issue with mypy, although it is just a workaround for mypy.
class Base:
def fun1(self, a: str):
pass
Your first example is unfortunately legitimately unsafe — it’s violating something known as the “Liskov substitution principle”.
To demonstrate why this is the case, let me simplify your example a little bit: I’ll have the base class accept any kind of object and have the child derived class accept an int. I also added in a little bit of runtime logic: the Base class just prints out the argument; the Derived class adds the argument against some arbitrary int.
class Base:
def fun(self, a: object) -> None:
print("Inside Base", a)
class Derived(Base):
def fun(self, a: int) -> None:
print("Inside Derived", a + 10)
On the surface, this seems perfectly fine. What could go wrong?
Well, suppose we write the following snippet. This snippet of code actually type checks perfectly fine: Derived is a subclass of Base, so we can pass an instance of Derived into any program that accepts an instance of Base. And similarly, Base.fun can accept any object, so surely it should be safe to pass in a string?
def accepts_base(b: Base) -> None:
b.fun("hello!")
accepts_base(Base())
accepts_base(Derived())
You might be able to see where this is going — this program is actually unsafe and will crash at runtime! Specifically, the very last line is broken: we pass in an instance of Derived, and Derived’s fun method only accepts ints. It’ll then try adding together the string it receives with 10, and promptly crash with a TypeError.
This is why mypy prohibits you from narrowing the types of the arguments in a method you’re overwriting. If Derived is a subclass of Base, that means we should be able to substitute an instance of Derived in any place we use Base without breaking anything. This rule is specifically known as the Liskov substitution principle.
Narrowing the argument types prevents this from happening.
(As a note, the fact that mypy requires you to respect Liskov is actually pretty standard. Pretty much all statically-typed languages with subtyping do the same thing — Java, C#, C++… The only counter-example I’m aware of is Eiffel.)
We can potentially run into similar issues with your original example. To make this a little more obvious, let me rename some of your classes to be a little more realistic. Let’s suppose we’re trying to write some sort of SQL execution engine, and write something that looks like this:
from typing import NewType
class BaseSQLExecutor:
def execute(self, query: str) -> None: ...
SanitizedSQLQuery = NewType('SanitizedSQLQuery', str)
class PostgresSQLExecutor:
def execute(self, query: SanitizedSQLQuery) -> None: ...
Notice that this code is identical to your original example! The only thing that’s different is the names.
We can again run into similar runtime issues — suppose we used the above classes like so:
def run_query(executor: BaseSQLExecutor, query: str) -> None:
executor.execute(query)
run_query(PostgresSQLExecutor, "my nasty unescaped and dangerous string")
If this were allowed to typecheck, we’ve introduced a potential security vulnerability into our code! The invariant that PostgresSQLExecutor can only accept strings we’ve explicitly decided to mark as a “SanitizedSQLQuery” type is broken.
Now, to address your other question: why is the case that mypy stops complaining if we make Base instead accept an argument of type Any?
Well, this is because the Any type has a very special meaning: it represents a 100% fully dynamic type. When you say “variable X is of type Any”, you’re actually saying “I don’t want you to assume anything about this variable — and I want to be able to use this type however I want without you complaining!”
It’s in fact inaccurate to call Any the “broadest type possible”. In reality, it’s simultaneously both the most broadest type AND the most narrowest type possible. Every single type is a subtype of Any AND Any is a subtype of all other types. Mypy will always pick whichever stance results in no type checking errors.
In essence, it’s an escape hatch, a way of telling the type checker “I know better”. Whenever you give a variable type Any, you’re actually completely opting out of any type-checking on that variable, for better or for worse.
For more on this, see typing.Any vs object?.
Finally, what can you do about all of this?
Well, unfortunately, I’m not sure necessarily an easy way around this: you’re going to have to redesign your code. It’s fundamentally unsound, and there aren’t really any tricks that’s guaranteed to dig you out of this.
Exactly how you go about doing this depends on what exactly you’re trying to do. Perhaps you could do something with generics, as one user suggested. Or perhaps you could just rename one of the methods as another suggested. Or alternatively, you could modify Base.fun so it uses the same type as Derived.fun or vice-versa; you could make Derived no longer inherit from Base. It all really depends on the details of your exact circumstance.
And of course, if the situation genuinely is intractable, you could give up on type-checking in that corner of that codebase entirely and make Base.fun(…) accept Any (and accept that you might start running into runtime errors).
Having to consider these questions and redesign your code may seem like an inconvenient hassle — however, I personally think this is something to celebrate! Mypy successfully prevented you from accidentally introducing a bug into your code and is pushing you towards writing more robust code.
Here, it might simply suffice to use a union in your derived class and type narrow from there to be Liskov-compliant:
from __future__ import annotations
import typing
class Base:
def fun(self, a: str) -> None:
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: str | SomeType) -> None:
if isinstance(a, str):
Base.fun(a) # ``super()`` can also be used, if your
# classes support cooperative multiple
# inheritance
else:
# The derived class implementation
For more complicated scenarios, where you would need to capture relationships between arguments and return types, you can use typing.overload.
In this case, it is important to note that order matters: mypy uses a "pick the first match" rule, meaning it reads the class definition top-to-bottom and uses the first overloaded signature that matches the caller signature.
from __future__ import annotations
import typing
class Base:
def fun(self, a: str) -> None:
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
@typing.overload
def fun(self, a: SomeType) -> None: ...
@typing.overload
def fun(self, a: str) -> None: ...
def fun(self, a: str | SomeType) -> None:
# Here, you'll want to type narrow to determine
# whether `a` is a `str` or a `SomeType`
pass
The above checks on the mypy playground.
(I personally prefer to limit my usage of Generics to cases where I am creating data structures, like containers or binary trees. These should support multiple "generic" data types, and my little brain is better able to conceptualize the use case of Generics for data structures.
For Liskov and dynamic dispatch problems, I usually either union my types and type narrow within the implementation body or use @typing.overload for more complicated problems.)
Look at the example code (mypy_test.py):
import typing
class Base:
def fun(self, a: str):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
now mypy complains:
mypy mypy_test.py
mypy_test.py:10: error: Argument 1 of "fun" incompatible with supertype "Base"
In that case, how can I work with class hierarchy and be type safe?
Software versions:
mypy 0.650
Python 3.7.1
What I tried:
import typing
class Base:
def fun(self, a: typing.Type[str]):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
But it didn’t help.
One user commented: “Looks like you cannot narrow down accepted types in overridden method?”
But in that case, if I use the broadest type possible (typing.Any) in a base class signature it shoudn’t work either. But it does:
import typing
class Base:
def fun(self, a: typing.Any):
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: SomeType):
pass
No complains from mypy with the code directly above.
Use a generic class as follows:
from typing import Generic
from typing import NewType
from typing import TypeVar
BoundedStr = TypeVar('BoundedStr', bound=str)
class Base(Generic[BoundedStr]):
def fun(self, a: BoundedStr) -> None:
pass
SomeType = NewType('SomeType', str)
class Derived(Base[SomeType]):
def fun(self, a: SomeType) -> None:
pass
The idea is to define a base class, with a generic type. Now you want this generic type to be a subtype of str, hence the bound=str directive.
Then you define your type SomeType, and when you subclass Base, you specify what the generic type variable is: in this case it’s SomeType. Then mypy checks that SomeType is a subtype of str (since we stated that BoundedStr must be a bounded by str), and in this case mypy is happy.
Of course, mypy will complain if you defined SomeType = NewType('SomeType', int) and used it as the type variable for Base or, more generally, if you subclass Base[SomeTypeVariable] if SomeTypeVariable is not a subtype of str.
I read in a comment that you want to ditch mypy. Don’t! Instead, learn how types work; when you’re in situations where you feel mypy is against you, it’s very likely that there’s something you didn’t quite understand. In this case seek some help from other people instead of giving up!
Both functions have the same name, so just rename 1 of the functions. mypy will give you the same error if you do this:
class Derived(Base):
def fun(self, a: int):
Renaming fun to fun1 solves the issue with mypy, although it is just a workaround for mypy.
class Base:
def fun1(self, a: str):
pass
Your first example is unfortunately legitimately unsafe — it’s violating something known as the “Liskov substitution principle”.
To demonstrate why this is the case, let me simplify your example a little bit: I’ll have the base class accept any kind of object and have the child derived class accept an int. I also added in a little bit of runtime logic: the Base class just prints out the argument; the Derived class adds the argument against some arbitrary int.
class Base:
def fun(self, a: object) -> None:
print("Inside Base", a)
class Derived(Base):
def fun(self, a: int) -> None:
print("Inside Derived", a + 10)
On the surface, this seems perfectly fine. What could go wrong?
Well, suppose we write the following snippet. This snippet of code actually type checks perfectly fine: Derived is a subclass of Base, so we can pass an instance of Derived into any program that accepts an instance of Base. And similarly, Base.fun can accept any object, so surely it should be safe to pass in a string?
def accepts_base(b: Base) -> None:
b.fun("hello!")
accepts_base(Base())
accepts_base(Derived())
You might be able to see where this is going — this program is actually unsafe and will crash at runtime! Specifically, the very last line is broken: we pass in an instance of Derived, and Derived’s fun method only accepts ints. It’ll then try adding together the string it receives with 10, and promptly crash with a TypeError.
This is why mypy prohibits you from narrowing the types of the arguments in a method you’re overwriting. If Derived is a subclass of Base, that means we should be able to substitute an instance of Derived in any place we use Base without breaking anything. This rule is specifically known as the Liskov substitution principle.
Narrowing the argument types prevents this from happening.
(As a note, the fact that mypy requires you to respect Liskov is actually pretty standard. Pretty much all statically-typed languages with subtyping do the same thing — Java, C#, C++… The only counter-example I’m aware of is Eiffel.)
We can potentially run into similar issues with your original example. To make this a little more obvious, let me rename some of your classes to be a little more realistic. Let’s suppose we’re trying to write some sort of SQL execution engine, and write something that looks like this:
from typing import NewType
class BaseSQLExecutor:
def execute(self, query: str) -> None: ...
SanitizedSQLQuery = NewType('SanitizedSQLQuery', str)
class PostgresSQLExecutor:
def execute(self, query: SanitizedSQLQuery) -> None: ...
Notice that this code is identical to your original example! The only thing that’s different is the names.
We can again run into similar runtime issues — suppose we used the above classes like so:
def run_query(executor: BaseSQLExecutor, query: str) -> None:
executor.execute(query)
run_query(PostgresSQLExecutor, "my nasty unescaped and dangerous string")
If this were allowed to typecheck, we’ve introduced a potential security vulnerability into our code! The invariant that PostgresSQLExecutor can only accept strings we’ve explicitly decided to mark as a “SanitizedSQLQuery” type is broken.
Now, to address your other question: why is the case that mypy stops complaining if we make Base instead accept an argument of type Any?
Well, this is because the Any type has a very special meaning: it represents a 100% fully dynamic type. When you say “variable X is of type Any”, you’re actually saying “I don’t want you to assume anything about this variable — and I want to be able to use this type however I want without you complaining!”
It’s in fact inaccurate to call Any the “broadest type possible”. In reality, it’s simultaneously both the most broadest type AND the most narrowest type possible. Every single type is a subtype of Any AND Any is a subtype of all other types. Mypy will always pick whichever stance results in no type checking errors.
In essence, it’s an escape hatch, a way of telling the type checker “I know better”. Whenever you give a variable type Any, you’re actually completely opting out of any type-checking on that variable, for better or for worse.
For more on this, see typing.Any vs object?.
Finally, what can you do about all of this?
Well, unfortunately, I’m not sure necessarily an easy way around this: you’re going to have to redesign your code. It’s fundamentally unsound, and there aren’t really any tricks that’s guaranteed to dig you out of this.
Exactly how you go about doing this depends on what exactly you’re trying to do. Perhaps you could do something with generics, as one user suggested. Or perhaps you could just rename one of the methods as another suggested. Or alternatively, you could modify Base.fun so it uses the same type as Derived.fun or vice-versa; you could make Derived no longer inherit from Base. It all really depends on the details of your exact circumstance.
And of course, if the situation genuinely is intractable, you could give up on type-checking in that corner of that codebase entirely and make Base.fun(…) accept Any (and accept that you might start running into runtime errors).
Having to consider these questions and redesign your code may seem like an inconvenient hassle — however, I personally think this is something to celebrate! Mypy successfully prevented you from accidentally introducing a bug into your code and is pushing you towards writing more robust code.
Here, it might simply suffice to use a union in your derived class and type narrow from there to be Liskov-compliant:
from __future__ import annotations
import typing
class Base:
def fun(self, a: str) -> None:
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
def fun(self, a: str | SomeType) -> None:
if isinstance(a, str):
Base.fun(a) # ``super()`` can also be used, if your
# classes support cooperative multiple
# inheritance
else:
# The derived class implementation
For more complicated scenarios, where you would need to capture relationships between arguments and return types, you can use typing.overload.
In this case, it is important to note that order matters: mypy uses a "pick the first match" rule, meaning it reads the class definition top-to-bottom and uses the first overloaded signature that matches the caller signature.
from __future__ import annotations
import typing
class Base:
def fun(self, a: str) -> None:
pass
SomeType = typing.NewType('SomeType', str)
class Derived(Base):
@typing.overload
def fun(self, a: SomeType) -> None: ...
@typing.overload
def fun(self, a: str) -> None: ...
def fun(self, a: str | SomeType) -> None:
# Here, you'll want to type narrow to determine
# whether `a` is a `str` or a `SomeType`
pass
The above checks on the mypy playground.
(I personally prefer to limit my usage of Generics to cases where I am creating data structures, like containers or binary trees. These should support multiple "generic" data types, and my little brain is better able to conceptualize the use case of Generics for data structures.
For Liskov and dynamic dispatch problems, I usually either union my types and type narrow within the implementation body or use @typing.overload for more complicated problems.)