How does concurrent.futures.as_completed work? Question: I’m learning about python concurrency and I was introduced with the concept of futures. I read that as_completed() takes an iterable of futures and yields them as they are done. I want to know how it works internally. Is it yielding completed tasks (futures) immediately? A naive approach would be …
The workers in ThreadPoolExecutor is not really daemon Question: The thing I cannot figure out is that although ThreadPoolExecutor uses daemon workers, they will still run even if main thread exit. I can provide a minimal example in python3.6.4: import concurrent.futures import time def fn(): while True: time.sleep(5) print(“Hello”) thread_pool = concurrent.futures.ThreadPoolExecutor() thread_pool.submit(fn) while True: …
How to chain futures in a non-blocking manner? That is, how to use one future as an input in another future without blocking? Question: Using the below example, how can future2 use the result of future1 once future1 is complete (without blocking future3 from being submitted)? from concurrent.futures import ProcessPoolExecutor import time def wait(seconds): time.sleep(seconds) …
python concurrent.futures.ProcessPoolExecutor: Performance of .submit() vs .map() Question: I am using concurrent.futures.ProcessPoolExecutor to find the occurrence of a number from a number range. The intent is to investigate the amount of speed-up performance gained from concurrency. To benchmark performance, I have a control – a serial code to perform said task (shown below). I have …
Finding the cause of a BrokenProcessPool in python's concurrent.futures Question: In a nutshell I get a BrokenProcessPool exception when parallelizing my code with concurrent.futures. No further error is displayed. I want to find the cause of the error and ask for ideas of how to do that. Full problem I am using concurrent.futures to parallelize …
What is the difference between concurrent.futures and asyncio.futures? Question: To clarify the reason for this question: It is confusing to use two modules with the same name. What do they represent that makes them distinct? What task(s) can one solve that the other can’t and vice-versa? Asked By: sargas || Source Answers: From the docs: …
How do you kill Futures once they have started? Question: I am using the new concurrent.futures module (which also has a Python 2 backport) to do some simple multithreaded I/O. I am having trouble understanding how to cleanly kill tasks started using this module. Check out the following Python 2/3 script, which reproduces the behavior …
How does ThreadPoolExecutor().map differ from ThreadPoolExecutor().submit? Question: I was just very confused by some code that I wrote. I was surprised to discover that: with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor: results = list(executor.map(f, iterable)) and with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor: results = list(map(lambda x: executor.submit(f, x), iterable)) produce different results. The first one produces a list of whatever …
ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool Question: I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I’ve noticed that, almost with identical code, using the Pool from concurrent.futures is way slower than using multiprocessing.Pool. This is the version using multiprocessing: def hard_work(n): # Real hard work here pass if …
