Python3.10 decorator confusion: how to wrap a class to augment a class's __init__ e.g. to track function calls
Question:
I’m getting myself confused with python decorators. Yes, there are a lot of useful resources out there (and I have consulted them prior to posting)
e.g.
- How can I use named arguments in a decorator?
- add method to a class dynamically with decorator
- functools.update_wrapper
- object wrapper recipe
I am sure there are better ways to do what I am about to describe, that is because this is just a simple toy example to understand the properties not what I am actually trying to do.
Suppose you have a class
class foo:
def __init__(self):
pass
def call_me(self, arg):
return arg
and I want to extend foo‘s constructor to take in two keyword arguments and modify the call_me method e.g. the desired end class would be:
class foo:
def __init__(self, keep_track_bool=False, do_print_bool=True):
self.keep_track = keep_track
self.memory = []
...
def call_me(self, arg):
arg = self.old_call_me(arg)
if self.do_print:
print('hi')
if self.keep_track:
self.memory.append(arg)
return arg
def old_call_me(self, arg):
...
return arg
Ideally, I would like a decorator so that I can wrap a bunch of classes which I am assuming have a method call_me like
@keep_track(do_print=False) # <--- sets default state
class foo:
...
@keep_track(keep_track=True) # <--- sets default state
class bar:
...
foo_instance = foo(do_print=True) # <-- I can change behavior of instance
How can I achieve this?
I have tried defining functions inside the decorator and setting them with
setattr(cls, 'old_call_me', call_me)
def call_me(self, arg):
# see above
setattr(cls, 'call_me', call_me)
and using functools.wrap
I would appreciate the guidance in this matter.
Answers:
I can offer a very, very convoluted, hard to read and only intended for learning purposes idea…
You could leverage the fact that if you try to access an instance’s attribute, Python will first go look for it in the instance attributes (let’s say self.__dict__), and if it doesn’t find it there, it will try to find it in the instance’s class attributes (let’s say self.__class__.__dict__)
So you could have your decorator write the default into the class itself and then accept it as an optional keyword argument to your __init__ method (basically creating only sometimes the do_print instance attribute).
from functools import wraps
def my_decorator(*, do_print):
def my_decorator_inner(klass):
print(f"Outer: {do_print}")
@wraps(klass)
def wrapper(*args, **kwargs):
print(f"Inner: {klass} {args}, {kwargs}")
klass.do_print = do_print
return klass(*args, **kwargs)
return wrapper
return my_decorator_inner
@my_decorator(do_print=False)
class Foo:
def __init__(self, *, do_print=None):
if do_print is not None:
self.do_print = do_print
def call_me(self):
if self.do_print:
print('hi')
else:
print("nopes, I ain't printing")
f_no_print = Foo()
f_no_print.call_me()
f_print = Foo(do_print = True)
f_print.call_me()
Note that I added a couple of print statements that might help with what’s being passed to each function.
HOWEVER this solution is very, very convoluted and confusing when what it is really doing is just:
class Foo:
do_print = False
def __init__(self, *, do_print=None):
if do_print is not None:
self.do_print = do_print
def call_me(self):
if self.do_print:
print('hi')
else:
print("nopes, I ain't printing")
f_no_print = Foo()
f_no_print.call_me()
f_print = Foo(do_print=True)
f_print.call_me()
EDIT as per comments below:
In Python everything is an object. Even functions are just… sort of """variables""". And classes are objects too. When you type class Foo:, you are not "just" creating a class definition (well… you are, but…) you are creating an instance called Foo whose type is type. You can alter that through metaclasses (which is where the __new__ method is usually used in). That object Foo of type type can then be called (inst = Foo()) to produce instances of type Foo
So, first things first. The cleaner subclassing method that @Tourelou was suggesting in his comment I imagine would go something like this:
class Foo:
def call_me(self, arg):
return arg
class FooPrinting(Foo):
def __init__(self, keep_track: bool = False, do_print: bool = True):
self.keep_track = keep_track
self.do_print = do_print
self.memory = []
def call_me(self, arg):
arg = super().call_me(arg)
if self.do_print:
print(f'hi {arg}')
if self.keep_track:
self.memory.append(arg)
return arg
f_no_print = Foo()
f_no_print.call_me(1)
f_print = FooPrinting(do_print=True)
f_print.call_me(2)
That’s good, that’s clean, that’s probably what you’d wanna use…
You mentioned in your comment that
Having only some instances have the keyword arg seems too non-pythonic
Actually, that is the Pythonic way. I understand it can look cluttered at first, but if you want to keep the ability of changing the behavior for some instances, you’ll have to accept the parameter in the constructor somehow. And kwargs with default values are a good way of doing it. True, true: you don’t really have to accept the parameters in the constructor, because you can dynamically add attributes to any instance at any point in your code… but you should. The fact you CAN do something doesn’t mean you SHOULD do it (this last line is also applicable to what comes below):
Now, since you seem to just want to sort of monkey-patch a method, there’s the option of dynamically swapping the class’ .call_me method by another method (new_call_me in the example below) that you have "kept in the freezer" and that you swap in the decorator:
from functools import wraps
def new_call_me(self, arg):
arg = self.old_call_me(arg)
if getattr(self, 'do_print', False):
print(f'swapped hi {arg}')
if getattr(self, 'keep_track', False):
self.memory.append(arg)
return arg
def my_decorator(*, do_print):
def my_decorator_inner(klass):
print(f"Outer: {do_print}")
@wraps(klass)
def wrapper(*args, **kwargs):
print(f"Inner: {klass} {args}, {kwargs}")
klass.do_print = do_print
if (not (hasattr(klass, 'old_call_me')) # <- If already swapped, skip
and hasattr(klass, 'call_me') and callable(klass.call_me)):
print(f"Swapping method in class {klass.__name__}")
klass.old_call_me = klass.call_me
klass.call_me = new_call_me
return klass(*args, **kwargs)
return wrapper
return my_decorator_inner
@my_decorator(do_print=True)
class Foo:
def __init__(self, *, do_print=None):
self.do_print = do_print
def call_me(self, arg):
return arg
f_no_print = Foo()
f_no_print.call_me(1)
f_print = Foo(do_print=True)
f_print.call_me(2)
⚠️ HOWEVER: If I ever see this in my company’s codebase, I’d yell. And if I have any saying in the matter, I’d try to get the author of such aberration fired. Ok, fine, FINE!… maybe not so much, ‘cuz I am a softie at heart , but for sure there’d be a conversation (a CON-VER-SA-TION!) with that person.
I’m getting myself confused with python decorators. Yes, there are a lot of useful resources out there (and I have consulted them prior to posting)
e.g.
- How can I use named arguments in a decorator?
- add method to a class dynamically with decorator
- functools.update_wrapper
- object wrapper recipe
I am sure there are better ways to do what I am about to describe, that is because this is just a simple toy example to understand the properties not what I am actually trying to do.
Suppose you have a class
class foo:
def __init__(self):
pass
def call_me(self, arg):
return arg
and I want to extend foo‘s constructor to take in two keyword arguments and modify the call_me method e.g. the desired end class would be:
class foo:
def __init__(self, keep_track_bool=False, do_print_bool=True):
self.keep_track = keep_track
self.memory = []
...
def call_me(self, arg):
arg = self.old_call_me(arg)
if self.do_print:
print('hi')
if self.keep_track:
self.memory.append(arg)
return arg
def old_call_me(self, arg):
...
return arg
Ideally, I would like a decorator so that I can wrap a bunch of classes which I am assuming have a method call_me like
@keep_track(do_print=False) # <--- sets default state
class foo:
...
@keep_track(keep_track=True) # <--- sets default state
class bar:
...
foo_instance = foo(do_print=True) # <-- I can change behavior of instance
How can I achieve this?
I have tried defining functions inside the decorator and setting them with
setattr(cls, 'old_call_me', call_me)
def call_me(self, arg):
# see above
setattr(cls, 'call_me', call_me)
and using functools.wrap
I would appreciate the guidance in this matter.
I can offer a very, very convoluted, hard to read and only intended for learning purposes idea…
You could leverage the fact that if you try to access an instance’s attribute, Python will first go look for it in the instance attributes (let’s say self.__dict__), and if it doesn’t find it there, it will try to find it in the instance’s class attributes (let’s say self.__class__.__dict__)
So you could have your decorator write the default into the class itself and then accept it as an optional keyword argument to your __init__ method (basically creating only sometimes the do_print instance attribute).
from functools import wraps
def my_decorator(*, do_print):
def my_decorator_inner(klass):
print(f"Outer: {do_print}")
@wraps(klass)
def wrapper(*args, **kwargs):
print(f"Inner: {klass} {args}, {kwargs}")
klass.do_print = do_print
return klass(*args, **kwargs)
return wrapper
return my_decorator_inner
@my_decorator(do_print=False)
class Foo:
def __init__(self, *, do_print=None):
if do_print is not None:
self.do_print = do_print
def call_me(self):
if self.do_print:
print('hi')
else:
print("nopes, I ain't printing")
f_no_print = Foo()
f_no_print.call_me()
f_print = Foo(do_print = True)
f_print.call_me()
Note that I added a couple of print statements that might help with what’s being passed to each function.
HOWEVER this solution is very, very convoluted and confusing when what it is really doing is just:
class Foo:
do_print = False
def __init__(self, *, do_print=None):
if do_print is not None:
self.do_print = do_print
def call_me(self):
if self.do_print:
print('hi')
else:
print("nopes, I ain't printing")
f_no_print = Foo()
f_no_print.call_me()
f_print = Foo(do_print=True)
f_print.call_me()
EDIT as per comments below:
In Python everything is an object. Even functions are just… sort of """variables""". And classes are objects too. When you type class Foo:, you are not "just" creating a class definition (well… you are, but…) you are creating an instance called Foo whose type is type. You can alter that through metaclasses (which is where the __new__ method is usually used in). That object Foo of type type can then be called (inst = Foo()) to produce instances of type Foo
So, first things first. The cleaner subclassing method that @Tourelou was suggesting in his comment I imagine would go something like this:
class Foo:
def call_me(self, arg):
return arg
class FooPrinting(Foo):
def __init__(self, keep_track: bool = False, do_print: bool = True):
self.keep_track = keep_track
self.do_print = do_print
self.memory = []
def call_me(self, arg):
arg = super().call_me(arg)
if self.do_print:
print(f'hi {arg}')
if self.keep_track:
self.memory.append(arg)
return arg
f_no_print = Foo()
f_no_print.call_me(1)
f_print = FooPrinting(do_print=True)
f_print.call_me(2)
That’s good, that’s clean, that’s probably what you’d wanna use…
You mentioned in your comment that
Having only some instances have the keyword arg seems too non-pythonic
Actually, that is the Pythonic way. I understand it can look cluttered at first, but if you want to keep the ability of changing the behavior for some instances, you’ll have to accept the parameter in the constructor somehow. And kwargs with default values are a good way of doing it. True, true: you don’t really have to accept the parameters in the constructor, because you can dynamically add attributes to any instance at any point in your code… but you should. The fact you CAN do something doesn’t mean you SHOULD do it (this last line is also applicable to what comes below):
Now, since you seem to just want to sort of monkey-patch a method, there’s the option of dynamically swapping the class’ .call_me method by another method (new_call_me in the example below) that you have "kept in the freezer" and that you swap in the decorator:
from functools import wraps
def new_call_me(self, arg):
arg = self.old_call_me(arg)
if getattr(self, 'do_print', False):
print(f'swapped hi {arg}')
if getattr(self, 'keep_track', False):
self.memory.append(arg)
return arg
def my_decorator(*, do_print):
def my_decorator_inner(klass):
print(f"Outer: {do_print}")
@wraps(klass)
def wrapper(*args, **kwargs):
print(f"Inner: {klass} {args}, {kwargs}")
klass.do_print = do_print
if (not (hasattr(klass, 'old_call_me')) # <- If already swapped, skip
and hasattr(klass, 'call_me') and callable(klass.call_me)):
print(f"Swapping method in class {klass.__name__}")
klass.old_call_me = klass.call_me
klass.call_me = new_call_me
return klass(*args, **kwargs)
return wrapper
return my_decorator_inner
@my_decorator(do_print=True)
class Foo:
def __init__(self, *, do_print=None):
self.do_print = do_print
def call_me(self, arg):
return arg
f_no_print = Foo()
f_no_print.call_me(1)
f_print = Foo(do_print=True)
f_print.call_me(2)
⚠️ HOWEVER: If I ever see this in my company’s codebase, I’d yell. And if I have any saying in the matter, I’d try to get the author of such aberration fired. Ok, fine, FINE!… maybe not so much, ‘cuz I am a softie at heart , but for sure there’d be a conversation (a CON-VER-SA-TION!) with that person.