How to make a Python threaded program (with Locks) to run on multi-process? Question: I have a multi-threaded program, and I want to let the user choose how to run them, either in serial, multi threads or multi cores, at least at the top level. A runnable demo is shown below illustrating my program’s logic. …
Using Python, why would sys.exit(0) hang, and not properly close a program? Question: The bug I am running into is that sys.exit(0) is not properly closing my program. I use pyinstaller to make this program an exe. When I use the exe, the program has to be shutdown using task manager. (I am running the …
multi-processing issue: different results in parallel and serial computing Question: I run this script in serial and paralle way. The testing purpose is to compute squared sums of different array in parallel way and get the same result of the serial way. In the parallel version, the arrays A,B and C are passed as targets …
Parallelization of un-bzipping millions of files Question: I have millions of compressed .bz2 files which I need to uncompressed. Can uncompression be parallelized ? I have access to the server with many cpu cores for the purpose. I worked with the following code which is correct but it is extremely slow. import os, glob, bz2 …
Python multiprocessing Queue broken after worker kill Question: I made a program to simulate heavy mail management by using python multiprocessing Pool and Queue : from multiprocessing import Pool, Queue import time import uuid import os NB_WORKERS = 3 NB_MAILS_PER_5_SECONDS = 2 MAIL_MANAGEMENT_DURATION_SECONDS = 1 def list_new_mails_id(): for i in range(NB_MAILS_PER_5_SECONDS): # fake mailbox msg …
In Python ProcessPoolExecutor, do you need call shutdown after getting a BrokenProcessPool exception? Question: In Python ProcessPoolExecutor, do you need call shutdown after getting a BrokenProcessPool exception? Say I have something like this: pool = ProcessPoolExecutor(max_workers=1) try: return pool.submit( do_something, ).result() except concurrent.futures.process.BrokenProcessPool as pool_exc: pool = None return None Is it a bad idea …
Multiprocessing code behaves differently when commenting out one print statement Question: This is my code: from random import random from multiprocessing import Process from multiprocessing import Queue import time def new(shared_queue): print(‘Consumer: Running’, flush=True) while shared_queue.qsize() > 0: shared_queue.get() print(shared_queue.qsize()) print(‘Consumer: Done’, flush=True) if __name__ == ‘__main__’: start_time = time.time() # Init Queue queue = …
How to stop subprocesses that communicate with the main process through request and response queues? Question: I have a Python program that starts N subprocesses (clients) which send requests to and listen for responses from the main process (server). The interprocess communication uses pipes through multiprocessing.Queue objects according to the following scheme (one queue per …
Parallelize loop with "non-mapable" function in Python Question: I solve numerically integral equation on the time interval from 0 to Tmax with predefined step dT. I do it in the for loop: list_of_values = [] for i in range(dT,Tmax+dT,dT): func_at_t= my_fancy_solver(func_at_t) list_of_values.append(func_at_t) The function my_fancy_solver has one argument and inside it looks as follows: def …
MPIRE WorkerPool causes memory leak Question: I have a python module with a function that runs in an infinite loop. Within this function I create WorkerPool with the mpire library. I cannot use the standard multiprocessing library because I have to pass non-picklable objects to the worker functions (at least I have not yet found …
