MPIRE WorkerPool causes memory leak

Question:

I have a python module with a function that runs in an infinite loop. Within this function I create WorkerPool with the mpire library. I cannot use the standard multiprocessing library because I have to pass non-picklable objects to the worker functions (at least I have not yet found a viable solution to do so …)

The code looks something like this

def workerPoolFunction():
    mpirePool = mpire.WorkerPool(n_jobs=4)
    mpirePool.set_shared_objects(nonPicklableObject)
    mpirePool.set_keep_alive(True)

    while(True):
        results = mpirePool.map(workerFunction)

Due to the code being significantly slowed down if I restart the workers with each loop I keep the workers alive. Unfortunately this seems to cause a minimal memory leak (4 to 500 Bytes each 2 – 5 loops) that adds up over time.

I have already narrowed the leak origin down to the mpirePool.map function call and excluded that it it is caused by the code executed within the worker function.
An explicit gc.collect() call doesn’t solve the problem.

So I’m currently running out of ideas …

Asked By: Viktor Katzy

||

Answers:

The problem seemed to be burried within my system, not the library.

Simple solution was to upgrade Python from 3.7 to >=3.8 (see MPIRE Issue #73)

Answered By: Viktor Katzy