Passing a function that takes **kwargs into concurrent.futures.ThreadPoolExecutor.map
Question:
I’ve just begun learning concurrent programming in Python.
I created a function which takes one positional argument and then **kwargs.
Code:
def download_from_api(api_client_method, **kwargs):
try:
response = api_client_method(**kwargs)
return response
except api.exceptions.APIException as e:
print(f'{kwargs}: {api_client_method.__name__} - {e})
I’d like to pass this function to concurrent.futures.ThreadPoolExecutor().map()
object.
Below is the code i used for trying, passing everything as positional args
, but it didnt work:
these_should_be_passed_as_keyword_args = ['A', 'B']
n = len(these_should_be_passed_as_keyword_args)
functions = [api_client.relevant_method] * n
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(download_from_api, functions, these_should_be_passed_as_keyword_args)
Any help is really appreciated.
Thank you!
Answers:
There are some good examples here:
Pass multiple parameters to concurrent.futures.Executor.map?
Alternatively, I prefer using .submit()
Here are the docs, and here is a tutorial
As I cant execute your download_from_api function, here is a basic example with a function that just prints random generated kwargs:
import string
import random
import concurrent.futures as cf
def your_function(argument, **kwargs):
print(argument, kwargs)
random_kwargs = [{
''.join(random.choice(string.ascii_lowercase) for _ in range(4)):
''.join(random.choice(string.ascii_lowercase) for _ in range(4))
} for _ in range(10)]
with cf.ThreadPoolExecutor() as executor:
futures = [executor.submit(your_function, argument, **kwargs)
for argument, kwargs in enumerate(random_kwargs)]
I’ve just begun learning concurrent programming in Python.
I created a function which takes one positional argument and then **kwargs.
Code:
def download_from_api(api_client_method, **kwargs):
try:
response = api_client_method(**kwargs)
return response
except api.exceptions.APIException as e:
print(f'{kwargs}: {api_client_method.__name__} - {e})
I’d like to pass this function to concurrent.futures.ThreadPoolExecutor().map()
object.
Below is the code i used for trying, passing everything as positional args
, but it didnt work:
these_should_be_passed_as_keyword_args = ['A', 'B']
n = len(these_should_be_passed_as_keyword_args)
functions = [api_client.relevant_method] * n
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(download_from_api, functions, these_should_be_passed_as_keyword_args)
Any help is really appreciated.
Thank you!
There are some good examples here:
Pass multiple parameters to concurrent.futures.Executor.map?
Alternatively, I prefer using .submit()
Here are the docs, and here is a tutorial
As I cant execute your download_from_api function, here is a basic example with a function that just prints random generated kwargs:
import string
import random
import concurrent.futures as cf
def your_function(argument, **kwargs):
print(argument, kwargs)
random_kwargs = [{
''.join(random.choice(string.ascii_lowercase) for _ in range(4)):
''.join(random.choice(string.ascii_lowercase) for _ in range(4))
} for _ in range(10)]
with cf.ThreadPoolExecutor() as executor:
futures = [executor.submit(your_function, argument, **kwargs)
for argument, kwargs in enumerate(random_kwargs)]