Why do we have to use the __dunder__ methods instead of operators when calling via super?
Question:
Why do we have to use __getitem__ rather than the usual operator access?
class MyDict(dict):
def __getitem__(self, key):
return super()[key]
We get TypeError: 'super' object is not subscriptable.
Instead we must use super().__getitem__(key), but I never fully understood why – what exactly is it that prevented super being implemented in a way that would allow the operator access?
Subscriptable was just an example, I have the same question for __getattr__, __init__, etc.
The docs attempt to explain why, but I don’t understand it.
Answers:
Dunder methods must be defined on the class, not the instance. super() would need to have an implementation of every magic method in order to make this work. It’s not worth writing all that code and keeping it up-to-date with the language definition (e.g. the introduction of matrix multiplication in 3.5 created three new dunder methods), when you can just tell users to write out the dunder methods by hand. That uses normal method lookup, which can be easily emulated.
CPython’s bug tracker’s issue 805304, “super instances don’t support item assignment”, has Raymond Hettinger give a detailed explanation of perceived difficulties.
The reason this doesn’t work automatically is that such methods have to be defined on the class due to Python’s caching of methods, whilst the proxied methods are found at runtime.
He offers a patch that would give a subset of this functionality:
+ if (o->ob_type == &PySuper_Type) {
+ PyObject *result;
+ result = PyObject_CallMethod(o, "__setitem__", "(OO)", key, value);
+ if (result == NULL)
+ return -1;
+ Py_DECREF(result);
+ return 0;
+ }
+
so it is clearly possible.
However, he concludes
I’ve been thinking that this one could be left alone and just
document that super objects only do their magic upon
explicit attribute lookup.
Otherwise, fixing it completely involves combing Python for
every place that directly calls functions from the slots table,
and then adding a followup call using attribute lookup if the
slot is empty.
When it comes to functions like repr(obj), I think we want
the super object to identify itself rather than forwarding the
call to the target object’s __repr__() method.
The argument seems to be that if __dunder__ methods are proxied, then either __repr__ is proxied or there is an inconsistency between them. super(), thus, might not want to proxy such methods lest it gets too near the programmer’s equivalent of an uncanny valley.
What you ask can be done, and easily. For instance:
class dundersuper(super):
def __add__(self, other):
# this works, because the __getattribute__ method of super is over-ridden to search
# through the given object's mro instead of super's.
return self.__add__(other)
super = dundersuper
class MyInt(int):
def __add__(self, other):
return MyInt(super() + other)
i = MyInt(0)
assert type(i + 1) is MyInt
assert i + 1 == MyInt(1)
So the reason that super works with magic methods isn’t because it’s not possible. The reason must lie elsewhere. One reason is that doing so would violate the contract of equals (==). That is equals is, amongst other criteria, symmetric. This means that if a == b is true then b == a must also be true. That lands us in a tricky situation, where super(self, CurrentClass) == self, but self != super(self, CurrentClass) eg.
class dundersuper(super):
def __eq__(self, other):
return self.__eq__(other)
super = dundersuper
class A:
def self_is_other(self, other):
return super() == other # a.k.a. object.__eq__(self, other) or self is other
def __eq__(self, other):
"""equal if both of type A"""
return A is type(self) and A is type(other)
class B:
def self_is_other(self, other):
return other == super() # a.k.a object.__eq__(other, super()), ie. False
def __eq__(self, other):
return B is type(self) and B is type(other)
assert A() == A()
a = A()
assert a.self_is_other(a)
assert B() == B()
b = B()
assert b.self_is_other(b) # assertion fails
Another reason is that once super is done searching it’s given object’s mro, it then has to give itself a chance to provide the requested attribute – super objects are still an objects in their own right — we should be able to test for equality with other objects, ask for string representations, and introspect the object and class super is working with. This creates a problem if the dunder method is available on the super object, but not on object that the mutable object represents. For instance:
class dundersuper(super):
def __add__(self, other):
return self.__add__(other)
def __iadd__(self, other):
return self.__iadd__(other)
super = dundersuper
class MyDoubleList(list):
"""Working, but clunky example."""
def __add__(self, other):
return MyDoubleList(super() + 2 * other)
def __iadd__(self, other):
s = super()
s += 2 * other # can't assign to the result of a function, so we must assign
# the super object to a local variable first
return s
class MyDoubleTuple(tuple):
"""Broken example -- iadd creates infinite recursion"""
def __add__(self, other):
return MyDoubleTuple(super() + 2 * other)
def __iadd__(self, other):
s = super()
s += 2 * other
return s
With the list example the function __iadd__ could have been more simply written as
def __iadd__(self, other):
return super().__iadd__(other)
With the tuple example we fall into infinite recursion, this is because tuple.__iadd__ does not exist. Therefore when looking up the attribute __iadd__ on a super object the actual super object is checked for an __iadd__ attribute (which does exist). We get that method and call it, which starts the whole process again. If we’d not written an __iadd__ method on super and used super().__iadd__(other) then this would never have happened. Rather we’d get an error message about a super object not having the attribute __iadd__. Slightly cryptic, but less so than an infinite stack trace.
So the reason super doesn’t work with magic methods is that it creates more problems than it solves.
Why do we have to use __getitem__ rather than the usual operator access?
class MyDict(dict):
def __getitem__(self, key):
return super()[key]
We get TypeError: 'super' object is not subscriptable.
Instead we must use super().__getitem__(key), but I never fully understood why – what exactly is it that prevented super being implemented in a way that would allow the operator access?
Subscriptable was just an example, I have the same question for __getattr__, __init__, etc.
The docs attempt to explain why, but I don’t understand it.
Dunder methods must be defined on the class, not the instance. super() would need to have an implementation of every magic method in order to make this work. It’s not worth writing all that code and keeping it up-to-date with the language definition (e.g. the introduction of matrix multiplication in 3.5 created three new dunder methods), when you can just tell users to write out the dunder methods by hand. That uses normal method lookup, which can be easily emulated.
CPython’s bug tracker’s issue 805304, “super instances don’t support item assignment”, has Raymond Hettinger give a detailed explanation of perceived difficulties.
The reason this doesn’t work automatically is that such methods have to be defined on the class due to Python’s caching of methods, whilst the proxied methods are found at runtime.
He offers a patch that would give a subset of this functionality:
+ if (o->ob_type == &PySuper_Type) {
+ PyObject *result;
+ result = PyObject_CallMethod(o, "__setitem__", "(OO)", key, value);
+ if (result == NULL)
+ return -1;
+ Py_DECREF(result);
+ return 0;
+ }
+
so it is clearly possible.
However, he concludes
I’ve been thinking that this one could be left alone and just
document that super objects only do their magic upon
explicit attribute lookup.Otherwise, fixing it completely involves combing Python for
every place that directly calls functions from the slots table,
and then adding a followup call using attribute lookup if the
slot is empty.When it comes to functions like repr(obj), I think we want
the super object to identify itself rather than forwarding the
call to the target object’s __repr__() method.
The argument seems to be that if __dunder__ methods are proxied, then either __repr__ is proxied or there is an inconsistency between them. super(), thus, might not want to proxy such methods lest it gets too near the programmer’s equivalent of an uncanny valley.
What you ask can be done, and easily. For instance:
class dundersuper(super):
def __add__(self, other):
# this works, because the __getattribute__ method of super is over-ridden to search
# through the given object's mro instead of super's.
return self.__add__(other)
super = dundersuper
class MyInt(int):
def __add__(self, other):
return MyInt(super() + other)
i = MyInt(0)
assert type(i + 1) is MyInt
assert i + 1 == MyInt(1)
So the reason that super works with magic methods isn’t because it’s not possible. The reason must lie elsewhere. One reason is that doing so would violate the contract of equals (==). That is equals is, amongst other criteria, symmetric. This means that if a == b is true then b == a must also be true. That lands us in a tricky situation, where super(self, CurrentClass) == self, but self != super(self, CurrentClass) eg.
class dundersuper(super):
def __eq__(self, other):
return self.__eq__(other)
super = dundersuper
class A:
def self_is_other(self, other):
return super() == other # a.k.a. object.__eq__(self, other) or self is other
def __eq__(self, other):
"""equal if both of type A"""
return A is type(self) and A is type(other)
class B:
def self_is_other(self, other):
return other == super() # a.k.a object.__eq__(other, super()), ie. False
def __eq__(self, other):
return B is type(self) and B is type(other)
assert A() == A()
a = A()
assert a.self_is_other(a)
assert B() == B()
b = B()
assert b.self_is_other(b) # assertion fails
Another reason is that once super is done searching it’s given object’s mro, it then has to give itself a chance to provide the requested attribute – super objects are still an objects in their own right — we should be able to test for equality with other objects, ask for string representations, and introspect the object and class super is working with. This creates a problem if the dunder method is available on the super object, but not on object that the mutable object represents. For instance:
class dundersuper(super):
def __add__(self, other):
return self.__add__(other)
def __iadd__(self, other):
return self.__iadd__(other)
super = dundersuper
class MyDoubleList(list):
"""Working, but clunky example."""
def __add__(self, other):
return MyDoubleList(super() + 2 * other)
def __iadd__(self, other):
s = super()
s += 2 * other # can't assign to the result of a function, so we must assign
# the super object to a local variable first
return s
class MyDoubleTuple(tuple):
"""Broken example -- iadd creates infinite recursion"""
def __add__(self, other):
return MyDoubleTuple(super() + 2 * other)
def __iadd__(self, other):
s = super()
s += 2 * other
return s
With the list example the function __iadd__ could have been more simply written as
def __iadd__(self, other):
return super().__iadd__(other)
With the tuple example we fall into infinite recursion, this is because tuple.__iadd__ does not exist. Therefore when looking up the attribute __iadd__ on a super object the actual super object is checked for an __iadd__ attribute (which does exist). We get that method and call it, which starts the whole process again. If we’d not written an __iadd__ method on super and used super().__iadd__(other) then this would never have happened. Rather we’d get an error message about a super object not having the attribute __iadd__. Slightly cryptic, but less so than an infinite stack trace.
So the reason super doesn’t work with magic methods is that it creates more problems than it solves.