concurrent.futures

Testing concurrent.futures.TimeoutError and logging in a threaded function using Pytest

Testing concurrent.futures.TimeoutError and logging in a threaded function using Pytest Question: I’ve come across a testing challenge in my Python project and I’m hoping to get some insights from the community. I have a utility module containing a function threaded_execute which utilizes the concurrent.futures module to execute a function in a separate thread. If a …

Total answers: 1

Python Parallel Processing with ThreadPoolExecutor Gives Wrong Results with Keras Model

Python Parallel Processing with ThreadPoolExecutor Gives Wrong Results with Keras Model Question: I am using parallel processing using the concurrent.futures.ThreadPoolExecutor class to make multiple predictions using a Keras model for different sets of weights. But the Keras model predictions using parallel processing are not correct. This is a reproducible sample code that creates 10 sets …

Total answers: 1

Python concurrent futures large number of inputs sit idle

Python concurrent futures large number of inputs sit idle Question: I am processing large number of files (tens of millions) using Python’s concurrent.futures. Issuing a small number of inputs work fine, however when the input size increases, the processes just don’t start. Below code executes only when input size is small, e.g. 20_000. import concurrent …

Total answers: 1

Why I can't use multiprocessing.Queue with ProcessPoolExecutor?

Why I can't use multiprocessing.Queue with ProcessPoolExecutor? Question: When I run the below code: from concurrent.futures import ProcessPoolExecutor, as_completed from multiprocessing import Queue q = Queue() def my_task(x, queue): queue.put("Task Complete") return x with ProcessPoolExecutor() as executor: tasks = [executor.submit(my_task, i, q) for i in range(10)] for task in as_completed(tasks): print(task.result()) I get this error: …

Total answers: 1

How to pass additional arguments to a function when using ThreadPoolExecutor?

How to pass additional arguments to a function when using ThreadPoolExecutor? Question: I would like to read several png images by utilizing the ThreadPoolExecutor and cv2.imread. Problem is that I don’t know where to place cv2.IMREAD_UNCHANGED tag/argument to preserve alpha channel (transparency). The following code works but alpha channel is lost. Where should I place …

Total answers: 1

Why parallelized code with concurrent.futures is slower then regular code?

Why parallelized code with concurrent.futures is slower then regular code? Question: I tried parallelizing with concurrent.futures expecting that parallelized code will be faster. I made a dafault code to test the parallelization. It is not important what the code does. I’m mainly interested in the speed of the dafault code and parallelized code. All it …

Total answers: 1

Python ThreadPoolExecutor (concurrent.futures) memory leak

Python ThreadPoolExecutor (concurrent.futures) memory leak Question: Hello I’m trying to load a big list==list.txt and send it to Function==Do_something() with concurrent.futures.ThreadPoolExecutor The problem is that whatever I do, the memory gets heavy, At first I thought the reason is that i open list.txt into a variable as (list) and because of that i changed code …

Total answers: 1

ProcessPoolExecutor using map hang on large load

ProcessPoolExecutor using map hang on large load Question: Experiencing hangs running ProcessPoolExecutor on map, only on a relatively large load. The behaviour we see is that after about 1 minutes of hard working, job seems to hang: the CPU utilization drops sharply then becomes idle; the stack trace also seems to show the same portion …

Total answers: 1

Python, use a dict over a list please

Python, use a dict over a list please Question: I have this code and it works. I wrote it to use a list a while back and I am revisiting it and wondering if I can add url and red to a dict rather than a list. def starts_with_hash(child: str) -> bool: if not child.startswith(‘#’): …

Total answers: 1