How to implement virtual methods in Python?
Question:
I know virtual methods from PHP or Java.
How can they be implemented in Python?
Or have I to define an empty method in an abstract class and override it?
Answers:
Python methods are always virtual.
Sure, and you don’t even have to define a method in the base class. In Python methods are better than virtual – they’re completely dynamic, as the typing in Python is duck typing.
class Dog:
def say(self):
print "hau"
class Cat:
def say(self):
print "meow"
pet = Dog()
pet.say() # prints "hau"
another_pet = Cat()
another_pet.say() # prints "meow"
my_pets = [pet, another_pet]
for a_pet in my_pets:
a_pet.say()
Cat
and Dog
in Python don’t even have to derive from a common base class to allow this behavior – you gain it for free. That said, some programmers prefer to define their class hierarchies in a more rigid way to document it better and impose some strictness of typing. This is also possible – see for example the abc
standard module.
Actually, in version 2.6 python provides something called abstract base classes and you can explicitly set virtual methods like this:
from abc import ABCMeta
from abc import abstractmethod
...
class C:
__metaclass__ = ABCMeta
@abstractmethod
def my_abstract_method(self, ...):
It works very well, provided the class does not inherit from classes that already use metaclasses.
Python methods are always virtual
like Ignacio said yet
Somehow class inheritance may be a better approach to implement what you want.
class Animal:
def __init__(self,name,legs):
self.name = name
self.legs = legs
def getLegs(self):
return "{0} has {1} legs".format(self.name, self.legs)
def says(self):
return "I am an unknown animal"
class Dog(Animal): # <Dog inherits from Animal here (all methods as well)
def says(self): # <Called instead of Animal says method
return "I am a dog named {0}".format(self.name)
def somethingOnlyADogCanDo(self):
return "be loyal"
formless = Animal("Animal", 0)
rover = Dog("Rover", 4) #<calls initialization method from animal
print(formless.says()) # <calls animal say method
print(rover.says()) #<calls Dog says method
print(rover.getLegs()) #<calls getLegs method from animal class
Results should be:
I am an unknown animal
I am a dog named Rover
Rover has 4 legs
raise NotImplementedError()
: dynamic type checking
This is the recommended exception to raise on "pure virtual methods" of "abstract" base classes that don’t implement a method.
https://docs.python.org/3.5/library/exceptions.html#NotImplementedError says:
This exception is derived from RuntimeError
. In user defined base classes, abstract methods should raise this exception when they require derived classes to override the method.
As others said, this is mostly a documentation convention and is not required, but this way you get a more meaningful exception than a missing attribute error.
dynamic.py
class Base(object):
def virtualMethod(self):
raise NotImplementedError()
def usesVirtualMethod(self):
return self.virtualMethod() + 1
class Derived(Base):
def virtualMethod(self):
return 1
print Derived().usesVirtualMethod()
Base().usesVirtualMethod()
gives:
2
Traceback (most recent call last):
File "./dynamic.py", line 13, in <module>
Base().usesVirtualMethod()
File "./dynamic.py", line 6, in usesVirtualMethod
return self.virtualMethod() + 1
File "./dynamic.py", line 4, in virtualMethod
raise NotImplementedError()
NotImplementedError
typing.Protocol
: static type checking (Python 3.8)
Python 3.8 added typing.Protocol
which now allows us to also statically type check that a virtual method is implemented on a subclass.
protocol.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> str:
pass
def fly_fast(self) -> str:
return 'CanFly.fly_fast'
class Bird(CanFly):
def fly(self):
return 'Bird.fly'
def fly_fast(self):
return 'Bird.fly_fast'
class FakeBird(CanFly):
pass
assert Bird().fly() == 'Bird.fly'
assert Bird().fly_fast() == 'Bird.fly_fast'
# mypy error
assert FakeBird().fly() is None
# mypy error
assert FakeBird().fly_fast() == 'CanFly.fly_fast'
If we run this file, the asserts pass, as we didn’t add any dynamic typechecking:
python protocol.py
but if we typecheck if mypy
:
python -m pip install --user mypy
mypy protocol.py
we get an error as expected:
protocol.py:22: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol.py:24: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
It is a bit unfortunate however that the error checking only picks up the error on instantiation, and not at class definition.
typing.Protocol
counts methods as abstract when their body is "empty"
I’m not sure what they count as empty, but both all of the following count as empty:
pass
...
ellipsis object
raise NotImplementedError()
So the best possibility is likely:
protocol_empty.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> None:
raise NotImplementedError()
class Bird(CanFly):
def fly(self):
return None
class FakeBird(CanFly):
pass
Bird().fly()
FakeBird().fly()
which fails as desired:
protocol_empty.py:15: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol_empty.py:15: note: The following method was marked implicitly abstract because it has an empty function body: "fly". If it is not meant to be abstract, explicitly return None.
but if e.g. we replace the:
raise NotImplementedError()
with some random "non-empty" statement such as:
x = 1
then mypy
does not count them as virtual and gives no errors.
@abc.abstractmethod
: dynamic + static + documentation in one go
Previously mentioned at: https://stackoverflow.com/a/19316077/895245 but the metaclass
syntax changed in Python 3 to:
class C(metaclass=abc.ABCMeta):
instead of the Python 2:
class C:
__metaclass__=abc.ABCMeta
so now to use @abc.abstractmethod
which was previously mentioned at https://stackoverflow.com/a/19316077/895245 you need:
abc_cheat.py
import abc
class C(metaclass=abc.ABCMeta):
@abc.abstractmethod
def m(self, i):
pass
try:
c = C()
except TypeError:
pass
else:
assert False
Vs raise NotImplementedError
and Protocol
:
- disadvantage: more verbose
- advantage: does all of dynamic checks, static checks and shows up on documentation (see below)
https://peps.python.org/pep-0544 does mention both approaches in passing
E.g.:
abc_bad.py
#!/usr/bin/env python
import abc
class CanFly(metaclass=abc.ABCMeta):
'''
doc
'''
@abc.abstractmethod
def fly(self) -> str:
'''
doc
'''
pass
class Bird(CanFly):
'''
doc
'''
def fly(self):
'''
doc
'''
return 'Bird.fly'
class Bat(CanFly):
'''
doc
'''
pass
def send_mail(flyer: CanFly) -> str:
'''
doc
'''
return flyer.fly()
assert send_mail(Bird()) == 'Bird.fly'
assert send_mail(Bat()) == 'Bat.fly'
then:
mypy abc_bad.py
fails as desired with:
main.py:40: error: Cannot instantiate abstract class "Bat" with abstract attribute "fly"
Sphinx: make it show on the documentation
See: How to annotate a member as abstract in Sphinx documentation?
Of the methods mentioned above, only one shows up on the sphinx documentation output: @abc.abstractmethod
.
Outro
Bibiography:
- https://peps.python.org/pep-0544 the
typing.Protocol
PEP
- Is it possible to make abstract classes?
- What to use in replacement of an interface/protocol in python
Tested on Python 3.10.7, mypy 0.982, Ubuntu 21.10.
Something like a virtual method in C++ (calling method implementation of a derived class through a reference or pointer to the base class) doesn’t make sense in Python, as Python doesn’t have typing. (I don’t know how virtual methods work in Java and PHP though.)
But if by “virtual” you mean calling the bottom-most implementation in the inheritance hierarchy, then that’s what you always get in Python, as several answers point out.
Well, almost always…
As dplamp pointed out, not all methods in Python behave like that. Dunder method don’t. And I think that’s a not so well known feature.
Consider this artificial example
class A:
def prop_a(self):
return 1
def prop_b(self):
return 10 * self.prop_a()
class B(A):
def prop_a(self):
return 2
Now
>>> B().prop_b()
20
>>> A().prob_b()
10
However, consider this one
class A:
def __prop_a(self):
return 1
def prop_b(self):
return 10 * self.__prop_a()
class B(A):
def __prop_a(self):
return 2
Now
>>> B().prop_b()
10
>>> A().prob_b()
10
The only thing we’ve changes was making prop_a()
a dunder method.
A problem with the first behavior can be that you can’t change the behavior of prop_a()
in the derived class without impacting the behavior of prop_b()
. This very nice talk by Raymond Hettinger gives an example for a use case where this is inconvenient.
Python 3.6 introduced __init_subclass__
and this let you simply do this:
class A:
def method(self):
'''method needs to be overwritten'''
return NotImplemented
def __init_subclass__(cls):
if cls.method is A.method:
raise NotImplementedError(
'Subclass has not overwritten method {method}!')
The benefit of this solution is that you avoid the abc metaclass and give the user a direct imperative how to do it right. In addition to another answer here that raises NotImplementedError
when calling the method. This solution is checked on runtime and not only IF the user calls the method.
I know virtual methods from PHP or Java.
How can they be implemented in Python?
Or have I to define an empty method in an abstract class and override it?
Python methods are always virtual.
Sure, and you don’t even have to define a method in the base class. In Python methods are better than virtual – they’re completely dynamic, as the typing in Python is duck typing.
class Dog:
def say(self):
print "hau"
class Cat:
def say(self):
print "meow"
pet = Dog()
pet.say() # prints "hau"
another_pet = Cat()
another_pet.say() # prints "meow"
my_pets = [pet, another_pet]
for a_pet in my_pets:
a_pet.say()
Cat
and Dog
in Python don’t even have to derive from a common base class to allow this behavior – you gain it for free. That said, some programmers prefer to define their class hierarchies in a more rigid way to document it better and impose some strictness of typing. This is also possible – see for example the abc
standard module.
Actually, in version 2.6 python provides something called abstract base classes and you can explicitly set virtual methods like this:
from abc import ABCMeta
from abc import abstractmethod
...
class C:
__metaclass__ = ABCMeta
@abstractmethod
def my_abstract_method(self, ...):
It works very well, provided the class does not inherit from classes that already use metaclasses.
Python methods are always virtual
like Ignacio said yet
Somehow class inheritance may be a better approach to implement what you want.
class Animal:
def __init__(self,name,legs):
self.name = name
self.legs = legs
def getLegs(self):
return "{0} has {1} legs".format(self.name, self.legs)
def says(self):
return "I am an unknown animal"
class Dog(Animal): # <Dog inherits from Animal here (all methods as well)
def says(self): # <Called instead of Animal says method
return "I am a dog named {0}".format(self.name)
def somethingOnlyADogCanDo(self):
return "be loyal"
formless = Animal("Animal", 0)
rover = Dog("Rover", 4) #<calls initialization method from animal
print(formless.says()) # <calls animal say method
print(rover.says()) #<calls Dog says method
print(rover.getLegs()) #<calls getLegs method from animal class
Results should be:
I am an unknown animal
I am a dog named Rover
Rover has 4 legs
raise NotImplementedError()
: dynamic type checking
This is the recommended exception to raise on "pure virtual methods" of "abstract" base classes that don’t implement a method.
https://docs.python.org/3.5/library/exceptions.html#NotImplementedError says:
This exception is derived from
RuntimeError
. In user defined base classes, abstract methods should raise this exception when they require derived classes to override the method.
As others said, this is mostly a documentation convention and is not required, but this way you get a more meaningful exception than a missing attribute error.
dynamic.py
class Base(object):
def virtualMethod(self):
raise NotImplementedError()
def usesVirtualMethod(self):
return self.virtualMethod() + 1
class Derived(Base):
def virtualMethod(self):
return 1
print Derived().usesVirtualMethod()
Base().usesVirtualMethod()
gives:
2
Traceback (most recent call last):
File "./dynamic.py", line 13, in <module>
Base().usesVirtualMethod()
File "./dynamic.py", line 6, in usesVirtualMethod
return self.virtualMethod() + 1
File "./dynamic.py", line 4, in virtualMethod
raise NotImplementedError()
NotImplementedError
typing.Protocol
: static type checking (Python 3.8)
Python 3.8 added typing.Protocol
which now allows us to also statically type check that a virtual method is implemented on a subclass.
protocol.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> str:
pass
def fly_fast(self) -> str:
return 'CanFly.fly_fast'
class Bird(CanFly):
def fly(self):
return 'Bird.fly'
def fly_fast(self):
return 'Bird.fly_fast'
class FakeBird(CanFly):
pass
assert Bird().fly() == 'Bird.fly'
assert Bird().fly_fast() == 'Bird.fly_fast'
# mypy error
assert FakeBird().fly() is None
# mypy error
assert FakeBird().fly_fast() == 'CanFly.fly_fast'
If we run this file, the asserts pass, as we didn’t add any dynamic typechecking:
python protocol.py
but if we typecheck if mypy
:
python -m pip install --user mypy
mypy protocol.py
we get an error as expected:
protocol.py:22: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol.py:24: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
It is a bit unfortunate however that the error checking only picks up the error on instantiation, and not at class definition.
typing.Protocol
counts methods as abstract when their body is "empty"
I’m not sure what they count as empty, but both all of the following count as empty:
pass
...
ellipsis objectraise NotImplementedError()
So the best possibility is likely:
protocol_empty.py
from typing import Protocol
class CanFly(Protocol):
def fly(self) -> None:
raise NotImplementedError()
class Bird(CanFly):
def fly(self):
return None
class FakeBird(CanFly):
pass
Bird().fly()
FakeBird().fly()
which fails as desired:
protocol_empty.py:15: error: Cannot instantiate abstract class "FakeBird" with abstract attribute "fly"
protocol_empty.py:15: note: The following method was marked implicitly abstract because it has an empty function body: "fly". If it is not meant to be abstract, explicitly return None.
but if e.g. we replace the:
raise NotImplementedError()
with some random "non-empty" statement such as:
x = 1
then mypy
does not count them as virtual and gives no errors.
@abc.abstractmethod
: dynamic + static + documentation in one go
Previously mentioned at: https://stackoverflow.com/a/19316077/895245 but the metaclass
syntax changed in Python 3 to:
class C(metaclass=abc.ABCMeta):
instead of the Python 2:
class C:
__metaclass__=abc.ABCMeta
so now to use @abc.abstractmethod
which was previously mentioned at https://stackoverflow.com/a/19316077/895245 you need:
abc_cheat.py
import abc
class C(metaclass=abc.ABCMeta):
@abc.abstractmethod
def m(self, i):
pass
try:
c = C()
except TypeError:
pass
else:
assert False
Vs raise NotImplementedError
and Protocol
:
- disadvantage: more verbose
- advantage: does all of dynamic checks, static checks and shows up on documentation (see below)
https://peps.python.org/pep-0544 does mention both approaches in passing
E.g.:
abc_bad.py
#!/usr/bin/env python
import abc
class CanFly(metaclass=abc.ABCMeta):
'''
doc
'''
@abc.abstractmethod
def fly(self) -> str:
'''
doc
'''
pass
class Bird(CanFly):
'''
doc
'''
def fly(self):
'''
doc
'''
return 'Bird.fly'
class Bat(CanFly):
'''
doc
'''
pass
def send_mail(flyer: CanFly) -> str:
'''
doc
'''
return flyer.fly()
assert send_mail(Bird()) == 'Bird.fly'
assert send_mail(Bat()) == 'Bat.fly'
then:
mypy abc_bad.py
fails as desired with:
main.py:40: error: Cannot instantiate abstract class "Bat" with abstract attribute "fly"
Sphinx: make it show on the documentation
See: How to annotate a member as abstract in Sphinx documentation?
Of the methods mentioned above, only one shows up on the sphinx documentation output: @abc.abstractmethod
.
Outro
Bibiography:
- https://peps.python.org/pep-0544 the
typing.Protocol
PEP - Is it possible to make abstract classes?
- What to use in replacement of an interface/protocol in python
Tested on Python 3.10.7, mypy 0.982, Ubuntu 21.10.
Something like a virtual method in C++ (calling method implementation of a derived class through a reference or pointer to the base class) doesn’t make sense in Python, as Python doesn’t have typing. (I don’t know how virtual methods work in Java and PHP though.)
But if by “virtual” you mean calling the bottom-most implementation in the inheritance hierarchy, then that’s what you always get in Python, as several answers point out.
Well, almost always…
As dplamp pointed out, not all methods in Python behave like that. Dunder method don’t. And I think that’s a not so well known feature.
Consider this artificial example
class A:
def prop_a(self):
return 1
def prop_b(self):
return 10 * self.prop_a()
class B(A):
def prop_a(self):
return 2
Now
>>> B().prop_b()
20
>>> A().prob_b()
10
However, consider this one
class A:
def __prop_a(self):
return 1
def prop_b(self):
return 10 * self.__prop_a()
class B(A):
def __prop_a(self):
return 2
Now
>>> B().prop_b()
10
>>> A().prob_b()
10
The only thing we’ve changes was making prop_a()
a dunder method.
A problem with the first behavior can be that you can’t change the behavior of prop_a()
in the derived class without impacting the behavior of prop_b()
. This very nice talk by Raymond Hettinger gives an example for a use case where this is inconvenient.
Python 3.6 introduced __init_subclass__
and this let you simply do this:
class A:
def method(self):
'''method needs to be overwritten'''
return NotImplemented
def __init_subclass__(cls):
if cls.method is A.method:
raise NotImplementedError(
'Subclass has not overwritten method {method}!')
The benefit of this solution is that you avoid the abc metaclass and give the user a direct imperative how to do it right. In addition to another answer here that raises NotImplementedError
when calling the method. This solution is checked on runtime and not only IF the user calls the method.