All example concurrent.futures code is failing with "BrokenProcessPool"

Question

I am trying to get a basic understanding of this before I create the actual application I need. I recently moved over from 2.7 to 3.3.

A direct copy-paste of this code from the python docs fails, as does a slightly simpler example from here.

This is my code, derived from the second example:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x

# Make sure the map and function are working
print([val for val in map(f, nums)])

# Test to make sure concurrent map is working
with concurrent.futures.ProcessPoolExecutor() as executor:
    for item in executor.map(f, nums):
        print(item)

And this is the output:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Traceback (most recent call last):
  File "<string>", line 420, in run_nodebug
  File "<module1>", line 13, in <module>
  File "C:Python33libconcurrentfutures_base.py", line 546, in result_iterator
    yield future.result()
  File "C:Python33libconcurrentfutures_base.py", line 399, in result
    return self.__get_result()
  File "C:Python33libconcurrentfutures_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

How can I get this code to work as expected? I was hoping the examples would just work out of the box.

Asked By: Gareth Webber

||

Source

Answer 1

This was my fault, for two reasons:

The code was un-guarded, i.e no if __name__
The strange looking Traceback was because the file was not saved. Never caused me an issue before, but did in this case.

Correcting both of those fixed the error.

Final test code:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x
def main():
    # Make sure the map and function are working
    print([val for val in map(f, nums)])

    # Test to make sure concurrent map is working
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print([val for val in executor.map(f, nums)])

if __name__ == '__main__':
    main()

Output, as expected:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Answered By: Gareth Webber

Answer 2

Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using processpoolexecutor or any other parallel code which spawns new processes.

Basically, all your code which creates new processes must be under if __name__ == '__main__': , for the same reason you cannot execute it in interpreter.

Answered By: rakesh

All example concurrent.futures code is failing with "BrokenProcessPool"

Question:

Answers: