How do I type hint a method with the type of the enclosing class?
Question:
I have the following code in Python 3:
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: Position) -> Position:
return Position(self.x + other.x, self.y + other.y)
But my editor (PyCharm) says that the reference Position can not be resolved (in the __add__ method). How should I specify that I expect the return type to be of type Position?
Edit: I think this is actually a PyCharm issue. It actually uses the information in its warnings, and code completion.
But correct me if I’m wrong, and need to use some other syntax.
Answers:
As of Python 3.11 (released in late 2022), there is available typing.Self designed for this purpose. Check PEP 673!
For previous Python versions, one had to consider that the name ‘Position’ is not available at the time the class body itself is parsed. I don’t know how you are using the type declarations, but Python’s PEP 484 – which is what most mode should use if using these typing hints say that you can simply put the name as a string at this point:
def __add__(self, other: 'Position') -> 'Position':
return Position(self.x + other.x, self.y + other.y)
Check the PEP 484 section on forward references – tools conforming to that will know to unwrap the class name from there and make use of it. (It is always important to have in mind that the Python language itself does nothing with these annotations. They are usually meant for static-code analysis, or one could have a library/framework for type-checking at runtime – but you have to explicitly set that.)
Update: Also, as of Python 3.7, check out PEP 563. As of Python 3.8, it is possible to write from __future__ import annotations to defer the evaluation of annotations. Forward-referencing classes should work straightforward.
Update 2: As of Python 3.10, PEP 563 is being retought, and it may be that PEP 649 is used in instead – it would simply allow the class name to be used, plain, without any quotes: the pep proposal is that it is resolved in a lazy way.
Update 3: As of Python 3.11, PEPs 563 and 649 to resolve forward references, mentioned above are still contending and it is likely none of them will go forward as it is now.
TL;DR: As of today (2019), in Python 3.7+ you can turn this feature on using a "future" statement, from __future__ import annotations.
(The behaviour enabled by from __future__ import annotations might become the default in future versions of Python, and was going to be made the default in Python 3.10. However, the change in 3.10 was reverted at the last minute, and now may not happen at all.)
In Python 3.6 or below, you should use a string.
I guess you got this exception:
NameError: name 'Position' is not defined
This is because Position must be defined before you can use it in an annotation, unless you are using Python with PEP 563 changes enabled.
Python 3.7+: from __future__ import annotations
Python 3.7 introduces PEP 563: postponed evaluation of annotations. A module that uses the future statement from __future__ import annotations will store annotations as strings automatically:
from __future__ import annotations
class Position:
def __add__(self, other: Position) -> Position:
...
This had been scheduled to become the default in Python 3.10, but this change has now been postponed. Since Python still is a dynamically typed language so no type-checking is done at runtime, typing annotations should have no performance impact, right? Wrong! Before Python 3.7, the typing module used to be one of the slowest python modules in core so for code that involves importing the typing module, you will see an up to 7 times increase in performance when you upgrade to 3.7.
Python <3.7: use a string
According to PEP 484, you should use a string instead of the class itself:
class Position:
...
def __add__(self, other: 'Position') -> 'Position':
...
If you use the Django framework, this may be familiar, as Django models also use strings for forward references (foreign key definitions where the foreign model is self or is not declared yet). This should work with Pycharm and other tools.
Sources
The relevant parts of PEP 484 and PEP 563, to spare you the trip:
Forward references
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
A situation where this occurs commonly is the definition of a container class, where the class being defined occurs in the signature of some of the methods. For example, the following code (the start of a simple binary tree implementation) does not work:
class Tree:
def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = right
To address this, we write:
class Tree:
def __init__(self, left: 'Tree', right: 'Tree'):
self.left = left
self.right = right
The string literal should contain a valid Python expression (i.e., compile(lit, ”, ‘eval’) should be a valid code object) and it should evaluate without errors once the module has been fully loaded. The local and global namespace in which it is evaluated should be the same namespaces in which default arguments to the same function would be evaluated.
and PEP 563:
Implementation
In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective __annotations__ dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation.
…
Enabling the future behavior in Python 3.7
The functionality described above can be enabled starting from Python 3.7 using the following special import:
from __future__ import annotations
Things that you may be tempted to do instead
A. Define a dummy Position
Before the class definition, place a dummy definition:
class Position(object):
pass
class Position(object):
...
This will get rid of the NameError and may even look OK:
>>> Position.__add__.__annotations__
{'other': __main__.Position, 'return': __main__.Position}
But is it?
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: False
other is Position: False
B. Monkey-patch in order to add the annotations:
You may want to try some Python metaprogramming magic and write a decorator
to monkey-patch the class definition in order to add annotations:
class Position:
...
def __add__(self, other):
return self.__class__(self.x + other.x, self.y + other.y)
The decorator should be responsible for the equivalent of this:
Position.__add__.__annotations__['return'] = Position
Position.__add__.__annotations__['other'] = Position
At least it seems right:
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: True
other is Position: True
Probably too much trouble.
Specifying the type as string is fine, but always grates me a bit that we are basically circumventing the parser. So you better not misspell any one of these literal strings:
def __add__(self, other: 'Position') -> 'Position':
return Position(self.x + other.x, self.y + other.y)
A slight variation is to use a bound typevar, at least then you have to write the string only once when declaring the typevar:
from typing import TypeVar
T = TypeVar('T', bound='Position')
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: T) -> T:
return Position(self.x + other.x, self.y + other.y)
When a string-based type hint is acceptable, the __qualname__ item can also be used. It holds the name of the class, and it is available in the body of the class definition.
class MyClass:
@classmethod
def make_new(cls) -> __qualname__:
return cls()
By doing this, renaming the class does not imply modifying the type hints. But I personally would not expect smart code editors to handle this form well.
If you only care about fixing the NameError: name 'Position' is not defined, you can either specify the class name as a string:
def __add__(self, other: 'Position') -> 'Position':
Or if you use Python 3.7 or higher, add the following line to the top of your code (just before the other imports)
from __future__ import annotations
However, if you also want this to work for subclasses, and return the specific subclass, you need to annotate the method as being a generic method, by using a TypeVar.
What is slightly uncommon is that the TypeVar is bound to the type of self. Basically, this typing hinting tells the type checker that the return type of __add__() and copy() are the same type as self.
from __future__ import annotations
from typing import TypeVar
T = TypeVar('T', bound=Position)
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self: T, other: Position) -> T:
return type(self)(self.x + other.x, self.y + other.y)
def copy(self: T) -> T:
return type(self)(self.x, self.y)
edit: @juanpa.arrivillaga brought to my attention a better way to do this; see https://stackoverflow.com/a/63237226
It’s recommended to do the above answer instead of this one below.
[old answer below, kept for posterity]
I ❤️ Paulo’s answer
However, there’s a point to be made about type hint inheritance in relation to self, which is that if you type hint by using a literal copy paste of the class name as a string, then your type hint won’t inherit in a correct or consistent way.
The solution to this is to provide return type hint by putting the type hint on the return in the function itself.
✅ For example, do this:
class DynamicParent:
def func(self):
# roundabout way of returning self in order to have inherited type hints of the return
# https://stackoverflow.com/a/64938978
_self:self.__class__ = self
return _self
❌ Instead of doing this:
class StaticParent:
def func(self) -> 'StaticParent':
return self
Below is the reason why you want to do the type hint via the roundabout ✅ way shown above
class StaticChild(StaticParent):
pass
class DynamicChild(DynamicParent):
pass
static_child = StaticChild()
dynamic_child = DynamicChild()
✅ dynamic_child screenshot shows that type hinting works correctly when referencing the self:
❌ static_child screenshot shows that type hinting is mistakenly pointing at the parent class, i.e. the type hint does not change correctly with inheritance; it is static because it will always point at the parent even when it should point at the child
Starting in Python 3.11 (to be released in late 2022), you’ll be able to use Self as the return type.
from typing import Self
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: Self) -> Self:
return type(self)(self.x + other.x, self.y + other.y)
Self is also included in the typing-extensions package (available on PyPi), which although not part of the standard library, is sort of a "preview" version of the typing module. Then you just import from typing_extensions instead of typing, e.g. from typing_extensions import Self. From https://pypi.org/project/typing-extensions/,
The typing_extensions module serves two related purposes:
- Enable use
of new type system features on older Python versions. For example,
typing.TypeGuard is new in Python 3.10, but typing_extensions allows
users on Python 3.6 through 3.9 to use it too.
- Enable experimentation
with new type system PEPs before they are accepted and added to the
typing module.
Currently, typing-extensions officially supports Python 3.7 and later.
Returning Self is often a good idea, but you must return an object of the same type as self, which means calling type(self) rather than Position.
from __future__ import annotations
import sys
if sys.version_info >= (3, 11):
from typing import Self
else:
from typing_extensions import Self
class Animal:
def __init__(self, name: str, says: str) -> None:
self.name = name
self.says = says
@classmethod
def from_description(cls, description: str = "|") -> Self:
descr = description.split("|")
return cls(descr[0], descr[1])
code from https://rednafi.github.io/reflections/self-type-in-python.html
I have the following code in Python 3:
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: Position) -> Position:
return Position(self.x + other.x, self.y + other.y)
But my editor (PyCharm) says that the reference Position can not be resolved (in the __add__ method). How should I specify that I expect the return type to be of type Position?
Edit: I think this is actually a PyCharm issue. It actually uses the information in its warnings, and code completion.
But correct me if I’m wrong, and need to use some other syntax.
As of Python 3.11 (released in late 2022), there is available typing.Self designed for this purpose. Check PEP 673!
For previous Python versions, one had to consider that the name ‘Position’ is not available at the time the class body itself is parsed. I don’t know how you are using the type declarations, but Python’s PEP 484 – which is what most mode should use if using these typing hints say that you can simply put the name as a string at this point:
def __add__(self, other: 'Position') -> 'Position':
return Position(self.x + other.x, self.y + other.y)
Check the PEP 484 section on forward references – tools conforming to that will know to unwrap the class name from there and make use of it. (It is always important to have in mind that the Python language itself does nothing with these annotations. They are usually meant for static-code analysis, or one could have a library/framework for type-checking at runtime – but you have to explicitly set that.)
Update: Also, as of Python 3.7, check out PEP 563. As of Python 3.8, it is possible to write from __future__ import annotations to defer the evaluation of annotations. Forward-referencing classes should work straightforward.
Update 2: As of Python 3.10, PEP 563 is being retought, and it may be that PEP 649 is used in instead – it would simply allow the class name to be used, plain, without any quotes: the pep proposal is that it is resolved in a lazy way.
Update 3: As of Python 3.11, PEPs 563 and 649 to resolve forward references, mentioned above are still contending and it is likely none of them will go forward as it is now.
TL;DR: As of today (2019), in Python 3.7+ you can turn this feature on using a "future" statement, from __future__ import annotations.
(The behaviour enabled by from __future__ import annotations might become the default in future versions of Python, and was going to be made the default in Python 3.10. However, the change in 3.10 was reverted at the last minute, and now may not happen at all.)
In Python 3.6 or below, you should use a string.
I guess you got this exception:
NameError: name 'Position' is not defined
This is because Position must be defined before you can use it in an annotation, unless you are using Python with PEP 563 changes enabled.
Python 3.7+: from __future__ import annotations
Python 3.7 introduces PEP 563: postponed evaluation of annotations. A module that uses the future statement from __future__ import annotations will store annotations as strings automatically:
from __future__ import annotations
class Position:
def __add__(self, other: Position) -> Position:
...
This had been scheduled to become the default in Python 3.10, but this change has now been postponed. Since Python still is a dynamically typed language so no type-checking is done at runtime, typing annotations should have no performance impact, right? Wrong! Before Python 3.7, the typing module used to be one of the slowest python modules in core so for code that involves importing the typing module, you will see an up to 7 times increase in performance when you upgrade to 3.7.
Python <3.7: use a string
According to PEP 484, you should use a string instead of the class itself:
class Position:
...
def __add__(self, other: 'Position') -> 'Position':
...
If you use the Django framework, this may be familiar, as Django models also use strings for forward references (foreign key definitions where the foreign model is self or is not declared yet). This should work with Pycharm and other tools.
Sources
The relevant parts of PEP 484 and PEP 563, to spare you the trip:
Forward references
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
A situation where this occurs commonly is the definition of a container class, where the class being defined occurs in the signature of some of the methods. For example, the following code (the start of a simple binary tree implementation) does not work:
class Tree: def __init__(self, left: Tree, right: Tree): self.left = left self.right = rightTo address this, we write:
class Tree: def __init__(self, left: 'Tree', right: 'Tree'): self.left = left self.right = rightThe string literal should contain a valid Python expression (i.e., compile(lit, ”, ‘eval’) should be a valid code object) and it should evaluate without errors once the module has been fully loaded. The local and global namespace in which it is evaluated should be the same namespaces in which default arguments to the same function would be evaluated.
and PEP 563:
Implementation
In Python 3.10, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective
__annotations__dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation.…
Enabling the future behavior in Python 3.7
The functionality described above can be enabled starting from Python 3.7 using the following special import:
from __future__ import annotations
Things that you may be tempted to do instead
A. Define a dummy Position
Before the class definition, place a dummy definition:
class Position(object):
pass
class Position(object):
...
This will get rid of the NameError and may even look OK:
>>> Position.__add__.__annotations__
{'other': __main__.Position, 'return': __main__.Position}
But is it?
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: False
other is Position: False
B. Monkey-patch in order to add the annotations:
You may want to try some Python metaprogramming magic and write a decorator
to monkey-patch the class definition in order to add annotations:
class Position:
...
def __add__(self, other):
return self.__class__(self.x + other.x, self.y + other.y)
The decorator should be responsible for the equivalent of this:
Position.__add__.__annotations__['return'] = Position
Position.__add__.__annotations__['other'] = Position
At least it seems right:
>>> for k, v in Position.__add__.__annotations__.items():
... print(k, 'is Position:', v is Position)
return is Position: True
other is Position: True
Probably too much trouble.
Specifying the type as string is fine, but always grates me a bit that we are basically circumventing the parser. So you better not misspell any one of these literal strings:
def __add__(self, other: 'Position') -> 'Position':
return Position(self.x + other.x, self.y + other.y)
A slight variation is to use a bound typevar, at least then you have to write the string only once when declaring the typevar:
from typing import TypeVar
T = TypeVar('T', bound='Position')
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: T) -> T:
return Position(self.x + other.x, self.y + other.y)
When a string-based type hint is acceptable, the __qualname__ item can also be used. It holds the name of the class, and it is available in the body of the class definition.
class MyClass:
@classmethod
def make_new(cls) -> __qualname__:
return cls()
By doing this, renaming the class does not imply modifying the type hints. But I personally would not expect smart code editors to handle this form well.
If you only care about fixing the NameError: name 'Position' is not defined, you can either specify the class name as a string:
def __add__(self, other: 'Position') -> 'Position':
Or if you use Python 3.7 or higher, add the following line to the top of your code (just before the other imports)
from __future__ import annotations
However, if you also want this to work for subclasses, and return the specific subclass, you need to annotate the method as being a generic method, by using a TypeVar.
What is slightly uncommon is that the TypeVar is bound to the type of self. Basically, this typing hinting tells the type checker that the return type of __add__() and copy() are the same type as self.
from __future__ import annotations
from typing import TypeVar
T = TypeVar('T', bound=Position)
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self: T, other: Position) -> T:
return type(self)(self.x + other.x, self.y + other.y)
def copy(self: T) -> T:
return type(self)(self.x, self.y)
edit: @juanpa.arrivillaga brought to my attention a better way to do this; see https://stackoverflow.com/a/63237226
It’s recommended to do the above answer instead of this one below.
[old answer below, kept for posterity]
I ❤️ Paulo’s answer
However, there’s a point to be made about type hint inheritance in relation to self, which is that if you type hint by using a literal copy paste of the class name as a string, then your type hint won’t inherit in a correct or consistent way.
The solution to this is to provide return type hint by putting the type hint on the return in the function itself.
✅ For example, do this:
class DynamicParent:
def func(self):
# roundabout way of returning self in order to have inherited type hints of the return
# https://stackoverflow.com/a/64938978
_self:self.__class__ = self
return _self
❌ Instead of doing this:
class StaticParent:
def func(self) -> 'StaticParent':
return self
Below is the reason why you want to do the type hint via the roundabout ✅ way shown above
class StaticChild(StaticParent):
pass
class DynamicChild(DynamicParent):
pass
static_child = StaticChild()
dynamic_child = DynamicChild()
✅ dynamic_child screenshot shows that type hinting works correctly when referencing the self:
❌ static_child screenshot shows that type hinting is mistakenly pointing at the parent class, i.e. the type hint does not change correctly with inheritance; it is static because it will always point at the parent even when it should point at the child
Starting in Python 3.11 (to be released in late 2022), you’ll be able to use Self as the return type.
from typing import Self
class Position:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def __add__(self, other: Self) -> Self:
return type(self)(self.x + other.x, self.y + other.y)
Self is also included in the typing-extensions package (available on PyPi), which although not part of the standard library, is sort of a "preview" version of the typing module. Then you just import from typing_extensions instead of typing, e.g. from typing_extensions import Self. From https://pypi.org/project/typing-extensions/,
The typing_extensions module serves two related purposes:
- Enable use
of new type system features on older Python versions. For example,
typing.TypeGuard is new in Python 3.10, but typing_extensions allows
users on Python 3.6 through 3.9 to use it too.- Enable experimentation
with new type system PEPs before they are accepted and added to the
typing module.
Currently, typing-extensions officially supports Python 3.7 and later.
Returning Self is often a good idea, but you must return an object of the same type as self, which means calling type(self) rather than Position.
from __future__ import annotations
import sys
if sys.version_info >= (3, 11):
from typing import Self
else:
from typing_extensions import Self
class Animal:
def __init__(self, name: str, says: str) -> None:
self.name = name
self.says = says
@classmethod
def from_description(cls, description: str = "|") -> Self:
descr = description.split("|")
return cls(descr[0], descr[1])
code from https://rednafi.github.io/reflections/self-type-in-python.html