Why doesn't spawned process start

Question

I’m having issues using Python’s multiprocessing module. Here is a (Very) simplified version of the issue i’m having trying to use a multiprocessing queue to communicate between the spawned process and the main process:

from time import sleep
import multiprocessing as mp

# add number to queue Q every 2 second, then None
def func(Q):
    for i in range(1,11):  
        Q.put(i)
        sleep(2)
    Q.put(None)

# start the process to add numbers
Q = mp.Queue()
P = mp.Process(target=func, args=(Q,))
P.start()

# place some initial values in the queue
for n in range(10,15):
    sleep(0.5)
    Q.put(n)

# wait a bit to let process start
print("waiting...")
sleep(2)

# get everything from the queue up to None
while True:
    n = Q.get()
    if n is None: break
    print(n)

The process (func) should be adding items to the queue but it is either not started or is getting stuck somehow.

The output i’m getting is:

waiting...
10
11
12
13
14

I would expect 1 through 10 to come out somewhere in there and the loop to eventually end (but it doesn’t)

What am I missing ?

By the way, I’m doing this on MacOS Monterey (12.6) with Python 3.7. on a Macbook Pro (2.9 GHz 6-Core Intel Core i9)

I also tried a bunch of simpler mutiprocessing examples from the net and none of them seem to work either (but most of them try to print from the child process which is not an issue I want/need to deal with).

SOLVED

I finally figured it out. MacOS only has "spawn" as the default start method from Python 3.8 (so my assumptions were wrong on that). Also I should have used the if __name__ == '__main__': condition in my oversimplified example because spawn actually starts a new interpreter and was executing the process creation code recursively. Introducing a delay in the feeding of the 11..14 numbers in the queue made it clearer that this was sufficient to make it work (I initially thought P.join() was the culprit).

Here is the code that works with my simplified example:

from time import sleep
import multiprocessing as mp

def func(Q):
    for i in range(1,11):  
        Q.put(i)
        sleep(1)
    Q.put(None)
    
if __name__ == "__main__":
    mp.set_start_method("spawn")
    
    # start the process to add numbers
    Q = mp.Queue()
    P = mp.Process(target=func, name="child", args=(Q,),daemon=True)
    P.start()    

    # place some initial values in the queue
    for n in range(10,15):
        sleep(0.5)
        Q.put(n)

    # wait a bit to let process start
    print("waiting...")
    sleep(2)

    # get everything from the queue up to None
    while True:
        n = Q.get()
        if n is None: break
        print(n)

Asked By: Alain T.

||

Source

Answer 1

Running your code, I get a big fat red warning that says

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
   freeze_support()
   ...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.

Once I implement as the instruction tells me, your code works fine.

I also did a little bit of refactoring to follow some PEP recommendations, just because I can.

from time import sleep
import multiprocessing as mp


def produce(queue: mp.Queue) -> None:
    for i in range(1, 11):
        queue.put(i)
        sleep(2)
    queue.put(None)


def consume(queue: mp.Queue) -> None:
    while True:
        n = queue.get()
        if n is None:
            break
        print(n)


if __name__ == '__main__':
    dataexchange = mp.Queue()
    for number in range(10, 15):
        dataexchange.put(number)
    process = mp.Process(target=produce, args=(dataexchange,))
    process.start()
    print("waiting...")
    sleep(2)

    consume(dataexchange)

Answered By: Thomas Weller

Answer 2

You don’t protect your code from import (necessary when using "spawn" as the start method), so the child process will raise a RuntimeError. The thing is that this exception is generated from the Child process, so the main process will continue to wait on responses from a child that has already exited. The error message may not be printed by IDE’s that don’t correctly redirect STDERR, so you should instead run your code via Terminal instead to make sure you see the error (and correct it as Thomas has done).

Answered By: Aaron

Why doesn't spawned process start

Question:

Answers: