Wait for all multiprocessing jobs to finish before continuing
Question:
I want to run a bunch of jobs in parallel and then continue once all the jobs are finished. I’ve got something like
# based on example code from https://pymotw.com/2/multiprocessing/basics.html
import multiprocessing
import random
import time
def worker(num):
"""A job that runs for a random amount of time between 5 and 10 seconds."""
time.sleep(random.randrange(5,11))
print('Worker:' + str(num) + ' finished')
return
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker, args=(i,))
jobs.append(p)
p.start()
# Iterate through the list of jobs and remove one that are finished, checking every second.
while len(jobs) > 0:
jobs = [job for job in jobs if job.is_alive()]
time.sleep(1)
print('*** All jobs finished ***')
it works, but I’m sure there must be a better way to wait for all the jobs to finish than iterating over them again and again until they are done.
Answers:
What about?
for job in jobs:
job.join()
This blocks until the first process finishes, then the next one and so on. See more about join()
You can make use of join.
It let you wait for another process to end.
t1 = Process(target=f, args=(x,))
t2 = Process(target=f, args=('bob',))
t1.start()
t2.start()
t1.join()
t2.join()
You can also use barrier It works as for threads, letting you specify a number of process you want to wait on and once this number is reached the barrier free them. Here client and server are asumed to be spawn as Process.
b = Barrier(2, timeout=5)
def server():
start_server()
b.wait()
while True:
connection = accept_connection()
process_server_connection(connection)
def client():
b.wait()
while True:
connection = make_connection()
process_client_connection(connection)
And if you want more functionalities like sharing data and more flow control you can use a manager.
I want to run a bunch of jobs in parallel and then continue once all the jobs are finished. I’ve got something like
# based on example code from https://pymotw.com/2/multiprocessing/basics.html
import multiprocessing
import random
import time
def worker(num):
"""A job that runs for a random amount of time between 5 and 10 seconds."""
time.sleep(random.randrange(5,11))
print('Worker:' + str(num) + ' finished')
return
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker, args=(i,))
jobs.append(p)
p.start()
# Iterate through the list of jobs and remove one that are finished, checking every second.
while len(jobs) > 0:
jobs = [job for job in jobs if job.is_alive()]
time.sleep(1)
print('*** All jobs finished ***')
it works, but I’m sure there must be a better way to wait for all the jobs to finish than iterating over them again and again until they are done.
What about?
for job in jobs:
job.join()
This blocks until the first process finishes, then the next one and so on. See more about join()
You can make use of join.
It let you wait for another process to end.
t1 = Process(target=f, args=(x,))
t2 = Process(target=f, args=('bob',))
t1.start()
t2.start()
t1.join()
t2.join()
You can also use barrier It works as for threads, letting you specify a number of process you want to wait on and once this number is reached the barrier free them. Here client and server are asumed to be spawn as Process.
b = Barrier(2, timeout=5)
def server():
start_server()
b.wait()
while True:
connection = accept_connection()
process_server_connection(connection)
def client():
b.wait()
while True:
connection = make_connection()
process_client_connection(connection)
And if you want more functionalities like sharing data and more flow control you can use a manager.