Why won't dynamically adding a `__call__` method to an instance work?
Question:
In both Python 2 and Python 3 the code:
class Foo(object):
pass
f = Foo()
f.__call__ = lambda *args : args
f(1, 2, 3)
returns as error Foo object is not callable. Why does that happen?
PS: With old-style classes it works as expected.
PPS: This behavior is intended (see accepted answer). As a work-around it’s possible to define a __call__ at class level that just forwards to another member and set this "normal" member to a per-instance __call__ implementation.
Python 3.10.8 (main, Nov 1 2022, 14:18:21) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: class Foo:
...: def __call__(self, *args, **kwargs):
...: return self.call(*args, **kwargs)
...:
In [2]: f = Foo()
In [3]: f.call = lambda *args, **kwargs: (args, kwargs)
In [4]: f(1,2,3)
Out[4]: ((1, 2, 3), {})
Answers:
Double-underscore methods are always looked up on the class, never the instance. See Special method lookup for new-style classes:
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
That’s because the type might need to support the same operation (in which case the special method is looked up on the metatype).
For example, classes are callable (that’s how you produce an instance), but if Python looked up the __call__ method on the actual object, then you could never do so on classes that implement __call__ for their instances. ClassObject() would become ClassObject.__call__() which would fail because the self parameter is not passed in to the unbound method. So instead type(ClassObject).__call__(ClassObject) is used, and calling instance() translates to type(instance).__call__(instance).
To work around this, you could add a __call__ method to the class which checks for a __call__ attribute on the class, and if there, calls it.
In new style classes (default 3.x) and inherited from object in 2.x the attribute interception methods __getattr__ and __getattribute__ are no longer called for built-in operations on dunder overloaded methods of instances and instead the search begins at the class.
The rationale behind this, involves a technicallity introduced by the presence of MetaClasses.
Because classes are instances of metaclasses and because metaclasses might define certain operations that act on classes, skipping the class (which in this case can be considered an instance) makes sense; you need to invoke the method defined in the metaclass which is defined to process classes (cls as a first argument). If instance look-up was used, the look-up would use the class method which is defined for instances of that class (self).
Another (disputed reason) involves optimization: Since built-in operations on instances are usually invoked very frequently, skipping the instance look-up altogether and going directly to the class saves us some time. [Source Lutz, Learning Python, 5th Edition]
The main area where this might be of inconvenience is when creating proxy objects that overload __getattr__ and __getattribute__ with intent to forward the calls to the embedded internal object. Since built-in invocation will skip the instance altogether they won’t get caught and as an effect won’t get forwarded to the embedded object.
An easy, but tedious, work-around for this is to actually overload all the dunders you wish to intercept in the proxy object. Then in these overloaded dunders you can delegate the call to the internal object as required.
The easiest work-around I can think of is setting the attribute on the class Foo with setattr:
setattr(Foo, '__call__', lambda *args: print(args))
f(1, 2, 3)
(<__main__.Foo object at 0x7f40640faa90>, 1, 2, 3)
In both Python 2 and Python 3 the code:
class Foo(object):
pass
f = Foo()
f.__call__ = lambda *args : args
f(1, 2, 3)
returns as error Foo object is not callable. Why does that happen?
PS: With old-style classes it works as expected.
PPS: This behavior is intended (see accepted answer). As a work-around it’s possible to define a __call__ at class level that just forwards to another member and set this "normal" member to a per-instance __call__ implementation.
Python 3.10.8 (main, Nov 1 2022, 14:18:21) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: class Foo:
...: def __call__(self, *args, **kwargs):
...: return self.call(*args, **kwargs)
...:
In [2]: f = Foo()
In [3]: f.call = lambda *args, **kwargs: (args, kwargs)
In [4]: f(1,2,3)
Out[4]: ((1, 2, 3), {})
Double-underscore methods are always looked up on the class, never the instance. See Special method lookup for new-style classes:
For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.
That’s because the type might need to support the same operation (in which case the special method is looked up on the metatype).
For example, classes are callable (that’s how you produce an instance), but if Python looked up the __call__ method on the actual object, then you could never do so on classes that implement __call__ for their instances. ClassObject() would become ClassObject.__call__() which would fail because the self parameter is not passed in to the unbound method. So instead type(ClassObject).__call__(ClassObject) is used, and calling instance() translates to type(instance).__call__(instance).
To work around this, you could add a __call__ method to the class which checks for a __call__ attribute on the class, and if there, calls it.
In new style classes (default 3.x) and inherited from object in 2.x the attribute interception methods __getattr__ and __getattribute__ are no longer called for built-in operations on dunder overloaded methods of instances and instead the search begins at the class.
The rationale behind this, involves a technicallity introduced by the presence of MetaClasses.
Because classes are instances of metaclasses and because metaclasses might define certain operations that act on classes, skipping the class (which in this case can be considered an instance) makes sense; you need to invoke the method defined in the metaclass which is defined to process classes (cls as a first argument). If instance look-up was used, the look-up would use the class method which is defined for instances of that class (self).
Another (disputed reason) involves optimization: Since built-in operations on instances are usually invoked very frequently, skipping the instance look-up altogether and going directly to the class saves us some time. [Source Lutz, Learning Python, 5th Edition]
The main area where this might be of inconvenience is when creating proxy objects that overload __getattr__ and __getattribute__ with intent to forward the calls to the embedded internal object. Since built-in invocation will skip the instance altogether they won’t get caught and as an effect won’t get forwarded to the embedded object.
An easy, but tedious, work-around for this is to actually overload all the dunders you wish to intercept in the proxy object. Then in these overloaded dunders you can delegate the call to the internal object as required.
The easiest work-around I can think of is setting the attribute on the class Foo with setattr:
setattr(Foo, '__call__', lambda *args: print(args))
f(1, 2, 3)
(<__main__.Foo object at 0x7f40640faa90>, 1, 2, 3)