Avoid deeply nested function calls Pythonically
Question:
I have a series of functions, where the result of one is fed into the next, plus other inputs:
result = function1(
function2(
function3(
function4(x, y, z),
a, b, c),
d, e, f),
g, h, i)
This is ugly and hard to understand. In particular it isn’t obvious that function1 is actually the last one to be called.
How can this code be written nicer, in a Pythonic way?
I could e.g. assign intermediate results to variables:
j = function4(x, y, z)
k = function3(j, a, b, c)
l = function2(k, d, e, f)
result = function1(l, g, h, i)
But this also puts additional variables for things I don’t need into the namespace, and may keep a large amount of memory from being freed – unless I add a del j, k, l, which is its own kind of ugly. Plus, I have to come up with names.
Or I could use the name of the final result also for the intermediate results:
result = function4(x, y, z)
result = function3(result, a, b, c)
result = function2(result, d, e, f)
result = function1(result, g, h, i)
The disadvantage here is that the same name is used for possibly wildly different things, which may make reading and debugging harder.
Then maybe _ or __?
__ = function4(x, y, z)
__ = function3(__, a, b, c)
__ = function2(__, d, e, f)
result = function1(__, g, h, i)
A bit better, but again not super clear. And I might have to add a del __ at the end.
Is there a better, more Pythonic way?
Answers:
I think nesting function calls and assigning results to variables is not a bad solution, especially if you capture the whole in a single function with descriptive name. The name of the function is self-documenting, allows for reuse of the complex structure, and the local variables created in the function only exist for the lifetime of the function execution, so your namespace remains clean.
Having said that, depending on the type of the values being passed around, you could subclass whatever that type is and add the functions as methods on the class, which would allow you to chain the calls, which I’d consider very Pythonic if the superclass lends itself to it.
An example of that is the way the pandas library works and has changed recently. In the past, many pandas.DataFrame methods would take an optional parameter called inplace to allow a user to change this:
import pandas as pd
df = pd.DataFrame(my_data)
df = df.some_operation(arg1, arg2)
df = df.some_other_operation(arg3, arg4)
...
To this:
import pandas as pd
df = pd.DataFrame(my_data)
df.some_operation(arg1, arg2, inplace=True)
df.some_other_operation(arg3, arg4, inplace=True)
With inplace=True, the original DataFrame would be modified (or at least, the end result would suggest that’s what happened), and in some cases it might still be returned, in other cases not at all. The idea being that it avoided the creation of additional data; also, many users would use multiple variables (df1, df2, etc.) to keep all the intermediate results around, often for no good reason.
However, as you may know that pattern is a bad idea in pandas for several reasons.
Instead, modern pandas methods consistently return the DataFrame itself, and inplace is being deprecated where it still exists. But because all functions now return the actual resulting DataFrame, this works:
import pandas as pd
df = pd.DataFrame(my_data)
.some_operation(arg1, arg2)
.some_other_operation(arg3, arg4)
(Note that this originally worked for many cases already in pandas, but consistency is not the point here, it’s the pattern that matters)
Your situation could be similar, depending on what you’re passing exactly. From the example you provided, it’s unclear what the types might be. But given that you consistently pass the function result as the first parameter, it seems likely that your implementation is something like:
def fun1(my_type_var, a, b, c):
# perform some operation on my_type_var and return result of same type
return operations_on(my_type_var, a, b, c)
def fun2(my_type_var, a, b, c):
# similar
return other_operations_on(my_type_var, a, b, c)
...
In that case, it would make more sense to:
class MyType:
def method1(self, a, b, c):
# perform some operation on self, then return the result, or self
mt = operations_on(self, a, b, c)
return mt
...
Because this would allow:
x = MyType().method1(arg1, arg2, arg3).method2(...)
And depending on what your type is exactly, you could choose to modify self and return that, or perform the operation with a new instance as a result like in the example above. What the better idea is depends on what you’re doing, how your type works internally, etc.
Fantastic answer by @Grismar
One more option to consider, if the functions are things you are creating, instead of transformations of data via, e.g., numpy or pandas or whatever, is to create functions where each function itself, on the surface, is only one level deep.
This would be in line with how Robert "Bob" Martin advises to write functions in his book, Clean Code.
The idea is that with each function, you descend one level of abstraction.
Lacking the context for what you’re doing, I can’t demonstrate this convincingly, i.e., with function names that actually make sense. But mechanically, here is how it would look:
result = make_result(a,b,c,i,j,k,x,y,z)
def make_result(a,b,c,i,j,k,x,y,z):
return function1(make_input_for_1(a,b,c,i,j,k,x,y,z))
def make_input_for_1(...):
return function2(make_input_for_2(...)
and so on and so forth.
This only makes sense if the nesting of the functions is indeed an ever-more-detailed implementation of the general task.
You can create a class that aids in chaining these functions together. The constructor can apply a higher-order function to each input function to store the output of the function as an instance field and return self, then set each of those wrapped functions as methods of the instance. Something like this:
import functools
class Chainable():
def __init__(self, funcs):
self.value = None
def wrapper(func):
@functools.wraps(func)
def run(*args, **kwargs):
if self.value is None:
self.value = func(*args, **kwargs)
return self
else:
self.value = func(self.value, *args, **kwargs)
return self
return run
for func in funcs:
setattr(self, func.__qualname__, wrapper(func))
def f1(a, b):
return a + b
def f2(a, b):
return a * b
res = Chainable([f1, f2]).f1(1, 2).f2(5).value
print(res)
I have a series of functions, where the result of one is fed into the next, plus other inputs:
result = function1(
function2(
function3(
function4(x, y, z),
a, b, c),
d, e, f),
g, h, i)
This is ugly and hard to understand. In particular it isn’t obvious that function1 is actually the last one to be called.
How can this code be written nicer, in a Pythonic way?
I could e.g. assign intermediate results to variables:
j = function4(x, y, z)
k = function3(j, a, b, c)
l = function2(k, d, e, f)
result = function1(l, g, h, i)
But this also puts additional variables for things I don’t need into the namespace, and may keep a large amount of memory from being freed – unless I add a del j, k, l, which is its own kind of ugly. Plus, I have to come up with names.
Or I could use the name of the final result also for the intermediate results:
result = function4(x, y, z)
result = function3(result, a, b, c)
result = function2(result, d, e, f)
result = function1(result, g, h, i)
The disadvantage here is that the same name is used for possibly wildly different things, which may make reading and debugging harder.
Then maybe _ or __?
__ = function4(x, y, z)
__ = function3(__, a, b, c)
__ = function2(__, d, e, f)
result = function1(__, g, h, i)
A bit better, but again not super clear. And I might have to add a del __ at the end.
Is there a better, more Pythonic way?
I think nesting function calls and assigning results to variables is not a bad solution, especially if you capture the whole in a single function with descriptive name. The name of the function is self-documenting, allows for reuse of the complex structure, and the local variables created in the function only exist for the lifetime of the function execution, so your namespace remains clean.
Having said that, depending on the type of the values being passed around, you could subclass whatever that type is and add the functions as methods on the class, which would allow you to chain the calls, which I’d consider very Pythonic if the superclass lends itself to it.
An example of that is the way the pandas library works and has changed recently. In the past, many pandas.DataFrame methods would take an optional parameter called inplace to allow a user to change this:
import pandas as pd
df = pd.DataFrame(my_data)
df = df.some_operation(arg1, arg2)
df = df.some_other_operation(arg3, arg4)
...
To this:
import pandas as pd
df = pd.DataFrame(my_data)
df.some_operation(arg1, arg2, inplace=True)
df.some_other_operation(arg3, arg4, inplace=True)
With inplace=True, the original DataFrame would be modified (or at least, the end result would suggest that’s what happened), and in some cases it might still be returned, in other cases not at all. The idea being that it avoided the creation of additional data; also, many users would use multiple variables (df1, df2, etc.) to keep all the intermediate results around, often for no good reason.
However, as you may know that pattern is a bad idea in pandas for several reasons.
Instead, modern pandas methods consistently return the DataFrame itself, and inplace is being deprecated where it still exists. But because all functions now return the actual resulting DataFrame, this works:
import pandas as pd
df = pd.DataFrame(my_data)
.some_operation(arg1, arg2)
.some_other_operation(arg3, arg4)
(Note that this originally worked for many cases already in pandas, but consistency is not the point here, it’s the pattern that matters)
Your situation could be similar, depending on what you’re passing exactly. From the example you provided, it’s unclear what the types might be. But given that you consistently pass the function result as the first parameter, it seems likely that your implementation is something like:
def fun1(my_type_var, a, b, c):
# perform some operation on my_type_var and return result of same type
return operations_on(my_type_var, a, b, c)
def fun2(my_type_var, a, b, c):
# similar
return other_operations_on(my_type_var, a, b, c)
...
In that case, it would make more sense to:
class MyType:
def method1(self, a, b, c):
# perform some operation on self, then return the result, or self
mt = operations_on(self, a, b, c)
return mt
...
Because this would allow:
x = MyType().method1(arg1, arg2, arg3).method2(...)
And depending on what your type is exactly, you could choose to modify self and return that, or perform the operation with a new instance as a result like in the example above. What the better idea is depends on what you’re doing, how your type works internally, etc.
Fantastic answer by @Grismar
One more option to consider, if the functions are things you are creating, instead of transformations of data via, e.g., numpy or pandas or whatever, is to create functions where each function itself, on the surface, is only one level deep.
This would be in line with how Robert "Bob" Martin advises to write functions in his book, Clean Code.
The idea is that with each function, you descend one level of abstraction.
Lacking the context for what you’re doing, I can’t demonstrate this convincingly, i.e., with function names that actually make sense. But mechanically, here is how it would look:
result = make_result(a,b,c,i,j,k,x,y,z)
def make_result(a,b,c,i,j,k,x,y,z):
return function1(make_input_for_1(a,b,c,i,j,k,x,y,z))
def make_input_for_1(...):
return function2(make_input_for_2(...)
and so on and so forth.
This only makes sense if the nesting of the functions is indeed an ever-more-detailed implementation of the general task.
You can create a class that aids in chaining these functions together. The constructor can apply a higher-order function to each input function to store the output of the function as an instance field and return self, then set each of those wrapped functions as methods of the instance. Something like this:
import functools
class Chainable():
def __init__(self, funcs):
self.value = None
def wrapper(func):
@functools.wraps(func)
def run(*args, **kwargs):
if self.value is None:
self.value = func(*args, **kwargs)
return self
else:
self.value = func(self.value, *args, **kwargs)
return self
return run
for func in funcs:
setattr(self, func.__qualname__, wrapper(func))
def f1(a, b):
return a + b
def f2(a, b):
return a * b
res = Chainable([f1, f2]).f1(1, 2).f2(5).value
print(res)