Python multiprocessing Pool 'raise ValueError("Pool not running") ValueError: Pool not running' function with return value

Question

I’m trying to run functions in parallel that has return value in a loop.
But it seem stuck at results = pool.map(algorithm_file.foo, population) in second iteration of the for loop with

    raise ValueError("Pool not running")
ValueError: Pool not running

Example code:

from multiprocessing.dummy import Pool
import algorithm_file

population = [1, 3, 4]
pool = Pool(len(population))

total = list()

for _ in range(10):
    results = pool.map(algorithm_file.foo, population)
    pool.close()
    pool.join()
    total.append(sum(results))

print(total)

Content inside algorithm_file.py

from random import randint

def foo(x):
    return x * randint(0,5)

I tried put pool = Pool(len(population)) in the for loop, but the program crashed middle way without exception trown.

I found some solution uses global list(). But is there anyway to make work with maintaining function with return value?

Python 3.7.3

Asked By: Zack0ne

||

Source

Answer 1

I think the issue is that once you close the pool, you cannot use it again. That’s why the first iteration goes through fine, but on the second one you get the "Pool not running" error.

So, one way to fix the provided snippet is to instantiate a new pool for each iteration:

for _ in range(10):
    pool = Pool(len(population))
    results = pool.map(algorithm_file.foo, population)
    pool.close()
    pool.join()
    total.append(sum(results))

However, note that it’s (IMO) more elegant and Pythonic to use the pool as a context manager, i.e.,

for _ in range(10):
    with Pool(len(population)) as pool:
        results = pool.map(algorithm_file.foo, population)
        total.append(sum(results))

Answered By: Moot

Python multiprocessing Pool 'raise ValueError("Pool not running") ValueError: Pool not running' function with return value

Question:

Answers: