How to apply a list of functions sequentially to a string using Python reduce or list comprehension?
Question:
Problem Statement
I would like to apply a list of functions fs = [ f, g, h ] sequentially to a string text=' abCdEf '
Something like f( g( h( text) ) ).
This could easily be accomplished with the following code:
# initial text
text = ' abCDef '
# list of functions to apply sequentially
fs = [str.rstrip, str.lstrip, str.lower]
for f in fs:
text = f(text)
# expected result is 'abcdef' with spaces stripped, and all lowercase
print(text)
Using functools.reduce
It seems that functools.reduce should do the job here, since it "consumes" the list of functions at each iteration.
from functools import reduce
# I know `reduce` requires two arguments, but I don't even know
# which one to chose as text of function from the list
reduce(f(text), fs)
# first interaction should call
y = str.rstrip(' abCDef ') --> ' abCDef'
# next iterations fails, because tries to call ' abCDef'() -- as a function
Unfortunately, this code doesn’t work, since each iteration returns a string istead of a function, and fails with TypeError : 'str' object is not callable.
QUESTION: Is there any solution using map, reduce or list comprehension to this problem?
Answers:
reduce can take three arguments:
reduce(function, iterable, initializer)
What are these three arguments in general?
function is a function of two arguments. Let’s call these two arguments t and f.- the first argument,
t, will start as initializer; then will continue as the return value of the previous call of function.
- the second argument,
f, is taken from iterable.
What are these three arguments in our case?
- the iterable is your list of function;
- the second argument
f is going to be one of the functions;
- the first argument
t must be the text;
- the initializer must be the initial text;
- the return of
function must be the resulting text;
function(t, f) must be f(t).
Finally:
from functools import reduce
# initial text
text = ' abCDef '
# list of functions to apply sequentially
fs = [str.rstrip, str.lstrip, str.lower]
result = reduce(lambda t,f: f(t), fs, text)
print(repr(result))
# 'abcdef'
You can try this:
import functools
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower]
text = functools.reduce(lambda store, func: func(store), fs, text)
print(text)
I think you have misunderstood how reduce works. Reduce reduces an iterable into a single value. The callback function can take two arguments, a store and a element.
The reduce function first creates a store variable. Then, looping through the iterable, it calls the function with the store variable and the current element, and updating the store to the returned value. Finally, the function returns the store value. The final argument is what the store variable starts with.
So in the snippet, it loops through the function array, and calls the respective function on it. The lambda will then return the processed value, updating the store.
Since you also asked for a map solution, here is one. My values contains single-element iterables, values[0] has the original value and values[i] has the value after applying the first i functions.
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower]
values = [[text]]
values += map(map, fs, values)
result = next(values[-1])
print(repr(result)) # prints 'abcdef'
But I wouldn’t recommend this. I was mostly curious whether I can do it. And now I’ll try to think of how to avoid building that auxiliary list.
Ok I found a way without that auxiliary list, taking only O(1) memory. It entangles maps with tee and chain to a self-feeding iterator and uses a deque to drive it and get the final element. Obviously just for fun. Here’s output of alternatingly running my above solution and the new solution three times each. It shows the result and the peak memory usage. I repeated the list 10000 times to drive the memory usage up.
result = 'abcdef' 3,126,760 bytes map_with_black_magic
result = 'abcdef' 12,972 bytes map_with_blacker_magic
result = 'abcdef' 3,031,048 bytes map_with_black_magic
result = 'abcdef' 12,476 bytes map_with_blacker_magic
result = 'abcdef' 3,031,048 bytes map_with_black_magic
result = 'abcdef' 8,052 bytes map_with_blacker_magic
Code (Attempt This Online!):
def map_with_black_magic(text, fs):
values = [[text]]
values += map(map, fs, values)
return next(values[-1])
def map_with_blacker_magic(text, fs):
parts = [[[text]]]
values = chain.from_iterable(parts)
it1, it2 = tee(values)
parts.append(map(list, map(map, fs, it1)))
return deque(it2, 1)[0][0]
from itertools import tee, chain
from collections import deque
import tracemalloc as tm
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower] * 10000
for func in [map_with_black_magic, map_with_blacker_magic] * 3:
tm.start()
result = func(text, fs)
memory = tm.get_traced_memory()[1]
tm.stop()
print(f'{result = !r}{memory:12,} bytes ', func.__name__)
Here’s an alternative solution, which allows you to compose any number of functions and save the composed function for reuse:
import functools as ft
def compose(*funcs):
return ft.reduce(lambda f, g: lambda x: f(g(x)), funcs)
Usage:
In [4]: strip_and_lower = compose(str.rstrip, str.lstrip, str.lower)
In [5]: strip_and_lower(' abCDef ')
Out[5]: 'abcdef'
In [6]: strip_and_lower(" AJWEGIAJWGIAWJWGIWAJ ")
Out[6]: 'ajwegiajwgiawjwgiwaj'
In [7]: strip_lower_title = compose(str.title, str.lower, str.strip)
In [8]: strip_lower_title(" hello world ")
Out[8]: 'Hello World'
Note that the order of functions matters; this works just like mathematical function composition, i.e., (f . g . h)(x) = f(g(h(x)) so the functions are applied from right to left.
Problem Statement
I would like to apply a list of functions fs = [ f, g, h ] sequentially to a string text=' abCdEf '
Something like f( g( h( text) ) ).
This could easily be accomplished with the following code:
# initial text
text = ' abCDef '
# list of functions to apply sequentially
fs = [str.rstrip, str.lstrip, str.lower]
for f in fs:
text = f(text)
# expected result is 'abcdef' with spaces stripped, and all lowercase
print(text)
Using functools.reduce
It seems that functools.reduce should do the job here, since it "consumes" the list of functions at each iteration.
from functools import reduce
# I know `reduce` requires two arguments, but I don't even know
# which one to chose as text of function from the list
reduce(f(text), fs)
# first interaction should call
y = str.rstrip(' abCDef ') --> ' abCDef'
# next iterations fails, because tries to call ' abCDef'() -- as a function
Unfortunately, this code doesn’t work, since each iteration returns a string istead of a function, and fails with TypeError : 'str' object is not callable.
QUESTION: Is there any solution using map, reduce or list comprehension to this problem?
reduce can take three arguments:
reduce(function, iterable, initializer)
What are these three arguments in general?
functionis a function of two arguments. Let’s call these two argumentstandf.- the first argument,
t, will start asinitializer; then will continue as the return value of the previous call offunction. - the second argument,
f, is taken fromiterable.
What are these three arguments in our case?
- the iterable is your list of function;
- the second argument
fis going to be one of the functions; - the first argument
tmust be the text; - the initializer must be the initial text;
- the return of
functionmust be the resulting text; function(t, f)must bef(t).
Finally:
from functools import reduce
# initial text
text = ' abCDef '
# list of functions to apply sequentially
fs = [str.rstrip, str.lstrip, str.lower]
result = reduce(lambda t,f: f(t), fs, text)
print(repr(result))
# 'abcdef'
You can try this:
import functools
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower]
text = functools.reduce(lambda store, func: func(store), fs, text)
print(text)
I think you have misunderstood how reduce works. Reduce reduces an iterable into a single value. The callback function can take two arguments, a store and a element.
The reduce function first creates a store variable. Then, looping through the iterable, it calls the function with the store variable and the current element, and updating the store to the returned value. Finally, the function returns the store value. The final argument is what the store variable starts with.
So in the snippet, it loops through the function array, and calls the respective function on it. The lambda will then return the processed value, updating the store.
Since you also asked for a map solution, here is one. My values contains single-element iterables, values[0] has the original value and values[i] has the value after applying the first i functions.
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower]
values = [[text]]
values += map(map, fs, values)
result = next(values[-1])
print(repr(result)) # prints 'abcdef'
But I wouldn’t recommend this. I was mostly curious whether I can do it. And now I’ll try to think of how to avoid building that auxiliary list.
Ok I found a way without that auxiliary list, taking only O(1) memory. It entangles maps with tee and chain to a self-feeding iterator and uses a deque to drive it and get the final element. Obviously just for fun. Here’s output of alternatingly running my above solution and the new solution three times each. It shows the result and the peak memory usage. I repeated the list 10000 times to drive the memory usage up.
result = 'abcdef' 3,126,760 bytes map_with_black_magic
result = 'abcdef' 12,972 bytes map_with_blacker_magic
result = 'abcdef' 3,031,048 bytes map_with_black_magic
result = 'abcdef' 12,476 bytes map_with_blacker_magic
result = 'abcdef' 3,031,048 bytes map_with_black_magic
result = 'abcdef' 8,052 bytes map_with_blacker_magic
Code (Attempt This Online!):
def map_with_black_magic(text, fs):
values = [[text]]
values += map(map, fs, values)
return next(values[-1])
def map_with_blacker_magic(text, fs):
parts = [[[text]]]
values = chain.from_iterable(parts)
it1, it2 = tee(values)
parts.append(map(list, map(map, fs, it1)))
return deque(it2, 1)[0][0]
from itertools import tee, chain
from collections import deque
import tracemalloc as tm
text = ' abCDef '
fs = [str.rstrip, str.lstrip, str.lower] * 10000
for func in [map_with_black_magic, map_with_blacker_magic] * 3:
tm.start()
result = func(text, fs)
memory = tm.get_traced_memory()[1]
tm.stop()
print(f'{result = !r}{memory:12,} bytes ', func.__name__)
Here’s an alternative solution, which allows you to compose any number of functions and save the composed function for reuse:
import functools as ft
def compose(*funcs):
return ft.reduce(lambda f, g: lambda x: f(g(x)), funcs)
Usage:
In [4]: strip_and_lower = compose(str.rstrip, str.lstrip, str.lower)
In [5]: strip_and_lower(' abCDef ')
Out[5]: 'abcdef'
In [6]: strip_and_lower(" AJWEGIAJWGIAWJWGIWAJ ")
Out[6]: 'ajwegiajwgiawjwgiwaj'
In [7]: strip_lower_title = compose(str.title, str.lower, str.strip)
In [8]: strip_lower_title(" hello world ")
Out[8]: 'Hello World'
Note that the order of functions matters; this works just like mathematical function composition, i.e., (f . g . h)(x) = f(g(h(x)) so the functions are applied from right to left.