python multiprocessing – Is it possible to use both Queue and Event simultaneously?

Question:

I tried to use multiprocessing Queue and Event in the way that seemed very straightforward to me, but actually didn’t work as I expected. My idea was to have a Queue object to pass tasks to workers and to use an Event object as a stop signal.

Here is a near-minimal example.

import multiprocessing as mp
from time import sleep


def worker(stop_flag: mp.Event, tasks: mp.Queue):
    while True:
        if stop_flag.is_set():
            print("Stop signal received")
            break
        # tasks.get()
        sleep(1)
    # tasks.close()
    # tasks.join_thread()


if __name__ == '__main__':
    tasks = mp.Queue()
    stop_flag = mp.Event()
    w = mp.Process(target=worker, args=(stop_flag, tasks))
    w.start()
    sleep(1)
    items = 2
    print("Starting work loop")
    for i in range(items):
        tasks.put(i)
        print(f"Put item #{i} of {items} in queue")
        sleep(2)
    stop_flag.set()
    print("Stop flag set")
    print("Stopping worker")
    w.join()
    print("Worker joined")
    w.close()
    print("Finished")

This code works OK (worker thread joins and process stops) if tasks.get() is commented out. But when I uncomment it, the worker process never gets a stop signal (condtion if stop_flag.is_set() never becomes true) and the main process hangs at w.join().

Of course, I can use a single Queue object for passing both work tasks and stop signals, but it would be much more complicated (work task is consumed by only one worker, but stop signal should be available to all of them, etc). So, the question is: is it possible to have both Queue and Event and how to do it correctly?

Asked By: Sergei Katkovsky

||
Source

Answers:

I find it helpful to split into worker, dispatcher and process management as its more obvious what is happening.

sleep is set so that the dispatcher completes and signals a stop before the worker has finished the tasks.

import multiprocessing as mp
from time import sleep

def worker(stop_flag: mp.Event, tasks: mp.Queue):
    while True:
        if stop_flag.is_set():
            print("Worker got stop")
            return
        task = tasks.get()
        print(task)
        sleep(0.2)


def dispatcher(stop_flag: mp.Event, tasks: mp.Queue):
    for task in range(10):
        tasks.put_nowait(task)
        sleep(0.1)
    stop_flag.set()


if __name__ == '__main__':
    queue = mp.Queue()
    flag = mp.Event()
    w = mp.Process(target=worker, args=(flag, queue))
    w.start()
    dispatcher(flag, queue)
    print('dispatcher has returned')
    w.join()
    print("worker has returned")

The output of this is:

0
1
2
3
4
dispatcher has returned
Worker got stop
worker has returned
Answered By: jwal

The issue you’re facing is because of the tasks.get() call that blocks the worker indefinitely when the queue is empty. The worker can’t check the stop flag if it’s blocked on the get() call.

You can use the timeout argument for the Queue.get() method. If the Queue.get() doesn’t receive an item within the specified timeout, it raises an Empty exception, and the code continues to the next iteration where it can check the stop_flag.

import multiprocessing as mp
from time import sleep
from queue import Empty


def worker(stop_flag: mp.Event, tasks: mp.Queue):
    while True:
        if stop_flag.is_set():
            print("Stop signal received")
            break
        try:
            task = tasks.get(timeout=1)  # <-- Add a timeout here
            # Process the task if needed
            print(f"Processed task: {task}")
        except Empty:
            pass  # No task received within the timeout
        sleep(1)  # You can remove this sleep if you don't need it anymore


if __name__ == '__main__':
    tasks = mp.Queue()
    stop_flag = mp.Event()
    w = mp.Process(target=worker, args=(stop_flag, tasks))
    w.start()
    sleep(1)
    items = 2
    print("Starting work loop")
    for i in range(items):
        tasks.put(i)
        print(f"Put item #{i} of {items} in queue")
        sleep(2)
    stop_flag.set()
    print("Stop flag set")
    print("Stopping worker")
    w.join()
    print("Worker joined")
    w.close()
    print("Finished")
Answered By: Yuri R

I generally avoid using Event objects unless their intended functionality is needed. If you never call Event.wait(), you’re not using it as a synchronization primitive, you’re just using it as a fancy boolean variable. That’s the case here. When I see an Event object declared, I take it as a hint that one Process will have to wait for something to happen in another Process.

Since you are already using a Queue, I prefer designating some special value as a sentinel to terminate the Queue. This way your Process will terminate cleanly and unambiguously, when the sentinel is received but not before. Here, None would work just fine.

import multiprocessing as mp
from time import sleep

def worker(tasks: mp.Queue):
    print("Worker", tasks)
    while True:
        t = tasks.get()
        print("Got", t)
        if t is None:
            break
        sleep(1.0)

if __name__ == '__main__':
    tasks = mp.Queue()
    w = mp.Process(target=worker, args=(tasks,))
    w.start()
    sleep(1.0)
    items = 2
    print("Starting work loop")
    for i in range(items):
        tasks.put(i)
        print(f"Put item #{i} of {items} in queue")
        sleep(2.0)
    tasks.put(None)
    print("Stop flag set")
    print("Stopping worker")
    w.join()
    print("Worker joined")
    w.close()
    print("Finished")

Sometimes you need to terminate the Process quickly, and the Queue has a bunch of items in it. You don’t want to wait for all of them to be processed before the sentinel gets handled. In that case, checking a boolean periodically makes sense. To do that, I would wrap a bool in a multiprocessing.Value object.

In many cases you will also need to have a timeout on tasks.get(), otherwise the secondary Process can hang waiting for the next item in an empty Queue.

import multiprocessing as mp
from time import sleep
from queue import Empty

def worker(a_bool: bool, tasks: mp.Queue):
    print("Worker", tasks)
    while not a_bool.value:
        try:
            t = tasks.get(timeout=0.1)
        except Empty:
            continue
        print("Got", t)
        if t is None:
            break
        sleep(1.0)

if __name__ == '__main__':
    tasks = mp.Queue()
    a_bool = mp.Value('b', False)
    w = mp.Process(target=worker, args=(a_bool, tasks))
    w.start()
    sleep(1.0)
    items = 2
    print("Starting work loop")
    for i in range(items):
        tasks.put(i)
        print(f"Put item #{i} of {items} in queue")
        sleep(2.0)
    a_bool.value = True
    print("Stop flag set")
    print("Stopping worker")
    w.join()
    print("Worker joined")
    w.close()
    print("Finished")
Answered By: Paul Cornelius