Multiprocess web application using API or MessageQueue

Question

I’m testing multiprocessing using apply_async.
However, it looks like each apply_async is called from MainProcess and it’s not exactly asynchronous. Each function is called only after previous one is finished. I’m not sure what I’m missing here.

I’m using Windows with Python 3.8, so it’s using the spawn method to create processes.

import os
import time
from multiprocessing import Pool, cpu_count, current_process
from threading import current_thread


def go_to_sleep():
    pid = os.getpid()
    thread_name = current_thread().name
    process_name = current_process().name
    print(f"{pid} Process {process_name} and {thread_name} going to sleep")
    time.sleep(5)


def apply_async():
    pool = Pool(processes=cpu_count())
    print(f"Number of procesess {len(pool._pool)}")
    for i in range(20):
        pool.apply_async(go_to_sleep())
    pool.close()
    pool.join()


def main():
    apply_async()


if __name__ == "__main__":
    start_time = time.perf_counter()
    main()
    end_time = time.perf_counter()
    print(f"Elapsed run time: {end_time - start_time} seconds.")

Output:

Number of procesess 8
26776 Process MainProcess and MainThread going to sleep
26776 Process MainProcess and MainThread going to sleep
26776 Process MainProcess and MainThread going to sleep

Asked By: chmtomasz

||

Source

Answer 1

The problem is that your code is not actually calling the specified function in the process pool, it is calling it in the main thread, and passing the result of calling it to pool.apply_async.

That is, instead of calling pool.apply_async(go_to_sleep()), you should call pool.apply_async(go_to_sleep). You need to pass the function that should be called to Pool.apply_async – you should not call the function when you call Pool.apply_async.

Answered By: dskrypa

Multiprocess web application using API or MessageQueue

Question:

Answers: