How does one monkey patch a function in python?
Question:
I’m having trouble replacing a function from a different module with another function and it’s driving me crazy.
Let’s say I have a module bar.py that looks like this:
from a_package.baz import do_something_expensive
def a_function():
print do_something_expensive()
And I have another module that looks like this:
from bar import a_function
a_function()
from a_package.baz import do_something_expensive
do_something_expensive = lambda: 'Something really cheap.'
a_function()
import a_package.baz
a_package.baz.do_something_expensive = lambda: 'Something really cheap.'
a_function()
I would expect to get the results:
Something expensive!
Something really cheap.
Something really cheap.
But instead I get this:
Something expensive!
Something expensive!
Something expensive!
What am I doing wrong?
Answers:
It may help to think of how Python namespaces work: they’re essentially dictionaries. So when you do this:
from a_package.baz import do_something_expensive
do_something_expensive = lambda: 'Something really cheap.'
think of it like this:
do_something_expensive = a_package.baz['do_something_expensive']
do_something_expensive = lambda: 'Something really cheap.'
Hopefully you can realize why this doesn’t work then 🙂 Once you import a name into a namespace, the value of the name in the namespace you imported from is irrelevant. You’re only modifying the value of do_something_expensive in the local module’s namespace, or in a_package.baz’s namespace, above. But because bar imports do_something_expensive directly, rather than referencing it from the module namespace, you need to write to its namespace:
import bar
bar.do_something_expensive = lambda: 'Something really cheap.'
There’s a really elegant decorator for this: Guido van Rossum: Python-Dev list: Monkeypatching Idioms.
There’s also the dectools package, which I saw an PyCon 2010, which may be able to be used in this context too, but that might actually go the other way (monkeypatching at the method declarative level… where you’re not)
In the first snippet, you make bar.do_something_expensive refer to the function object that a_package.baz.do_something_expensive refers to at that moment. To really “monkeypatch” that you would need to change the function itself (you are only changing what names refer to); this is possible, but you do not actually want to do that.
In your attempts to change the behavior of a_function, you have done two things:
-
In the first attempt, you make do_something_expensive a global name in your module. However, you are calling a_function, which does not look in your module to resolve names, so it still refers to the same function.
-
In the second example you change what a_package.baz.do_something_expensive refers to, but bar.do_something_expensive is not magically tied to it. That name still refers to the function object it looked up when it was initilized.
The simplest but far-from-ideal approach would be to change bar.py to say
import a_package.baz
def a_function():
print a_package.baz.do_something_expensive()
The right solution is probably one of two things:
- Redefine
a_function to take a function as an argument and call that, rather than trying to sneak in and change what function it is hard coded to refer to, or
- Store the function to be used in an instance of a class; this is how we do mutable state in Python.
Using globals (this is what changing module-level stuff from other modules is) is a bad thing that leads to unmaintainable, confusing, untestestable, unscalable code the flow of which is difficult to track.
do_something_expensive in the a_function() function is just a variable within the namespace of the module pointing to a function object. When you redefine the module you are doing it in a different namespace.
If you want to only patch it for your call and otherwise leave the original code you can use https://docs.python.org/3/library/unittest.mock.html#patch (since Python 3.3):
with patch('a_package.baz.do_something_expensive', new=lambda: 'Something really cheap.'):
print do_something_expensive()
# prints 'Something really cheap.'
print do_something_expensive()
# prints 'Something expensive!'
I’m having trouble replacing a function from a different module with another function and it’s driving me crazy.
Let’s say I have a module bar.py that looks like this:
from a_package.baz import do_something_expensive
def a_function():
print do_something_expensive()
And I have another module that looks like this:
from bar import a_function
a_function()
from a_package.baz import do_something_expensive
do_something_expensive = lambda: 'Something really cheap.'
a_function()
import a_package.baz
a_package.baz.do_something_expensive = lambda: 'Something really cheap.'
a_function()
I would expect to get the results:
Something expensive!
Something really cheap.
Something really cheap.
But instead I get this:
Something expensive!
Something expensive!
Something expensive!
What am I doing wrong?
It may help to think of how Python namespaces work: they’re essentially dictionaries. So when you do this:
from a_package.baz import do_something_expensive
do_something_expensive = lambda: 'Something really cheap.'
think of it like this:
do_something_expensive = a_package.baz['do_something_expensive']
do_something_expensive = lambda: 'Something really cheap.'
Hopefully you can realize why this doesn’t work then 🙂 Once you import a name into a namespace, the value of the name in the namespace you imported from is irrelevant. You’re only modifying the value of do_something_expensive in the local module’s namespace, or in a_package.baz’s namespace, above. But because bar imports do_something_expensive directly, rather than referencing it from the module namespace, you need to write to its namespace:
import bar
bar.do_something_expensive = lambda: 'Something really cheap.'
There’s a really elegant decorator for this: Guido van Rossum: Python-Dev list: Monkeypatching Idioms.
There’s also the dectools package, which I saw an PyCon 2010, which may be able to be used in this context too, but that might actually go the other way (monkeypatching at the method declarative level… where you’re not)
In the first snippet, you make bar.do_something_expensive refer to the function object that a_package.baz.do_something_expensive refers to at that moment. To really “monkeypatch” that you would need to change the function itself (you are only changing what names refer to); this is possible, but you do not actually want to do that.
In your attempts to change the behavior of a_function, you have done two things:
-
In the first attempt, you make do_something_expensive a global name in your module. However, you are calling
a_function, which does not look in your module to resolve names, so it still refers to the same function. -
In the second example you change what
a_package.baz.do_something_expensiverefers to, butbar.do_something_expensiveis not magically tied to it. That name still refers to the function object it looked up when it was initilized.
The simplest but far-from-ideal approach would be to change bar.py to say
import a_package.baz
def a_function():
print a_package.baz.do_something_expensive()
The right solution is probably one of two things:
- Redefine
a_functionto take a function as an argument and call that, rather than trying to sneak in and change what function it is hard coded to refer to, or - Store the function to be used in an instance of a class; this is how we do mutable state in Python.
Using globals (this is what changing module-level stuff from other modules is) is a bad thing that leads to unmaintainable, confusing, untestestable, unscalable code the flow of which is difficult to track.
do_something_expensive in the a_function() function is just a variable within the namespace of the module pointing to a function object. When you redefine the module you are doing it in a different namespace.
If you want to only patch it for your call and otherwise leave the original code you can use https://docs.python.org/3/library/unittest.mock.html#patch (since Python 3.3):
with patch('a_package.baz.do_something_expensive', new=lambda: 'Something really cheap.'):
print do_something_expensive()
# prints 'Something really cheap.'
print do_something_expensive()
# prints 'Something expensive!'