Wait for the first subprocess to finish
Question:
I have a list of subprocess‘ processes. I do not communicate with them and just wait.
I want to wait for the first process to finish (this solution works):
import subprocess
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
# wait for the first process to finish
while True:
over = False
for child in {a, b}:
try:
rst = child.wait(timeout=5)
except subprocess.TimeoutExpired:
continue # this subprocess is still running
if rst is not None: # subprocess is no more running
over = True
break # If either subprocess exits, so do we.
if over:
break
I don’t want use os.wait(), cause it could return from another subprocess not part of the list I’m waiting for.
A nice and elegant solution would probably be with an epoll or select and without any loop.
Answers:
There are pretty much 2 ways to do this, if you want the commands to be blocking and halt your program until it finishes, use subprocess.call
a = subprocess.call('...')
b = subprocess.call('...')
I don’t think this is what you want though.
If you do not want them to halt your entire program and only need to check if one of them is finished before calling another, you should use .communicate
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
....
for child in {a, b}:
try:
result, err = child.communicate(timeout=5)
.communicate is pretty much the most elegant, simple and recommended solution
If you don’t need to get output from the processes, Popen.poll() seems to be the simplest way to check whether they are done. The while True loop below is purely for demonstration purposes: you can decide how to do this in your larger program (eg, do the checking in a separate thread, do the checking in between the other work of the program, etc).
from subprocess import Popen
from time import sleep
ps = [
Popen(['sleep', t])
for t in ('3', '5', '2')
]
while True:
exit_codes = [p.poll() for p in ps]
print(exit_codes)
if any(ec is not None for ec in exit_codes):
break
else:
sleep(1)
Demo output:
[None, None, None]
[None, None, None]
[None, None, 0]
Here’s a solution using psutil – which is aimed exactly at this use-case:
import subprocess
import psutil
a = subprocess.Popen(['/bin/sleep', "2"])
b = subprocess.Popen(['/bin/sleep', "4"])
procs_list = [psutil.Process(a.pid), psutil.Process(b.pid)]
def on_terminate(proc):
print("process {} terminated".format(proc))
# waits for multiple processes to terminate
gone, alive = psutil.wait_procs(procs_list, timeout=3, callback=on_terminate)
Or, if you’d like to have a loop waiting for one of the process to be done:
while True:
gone, alive = psutil.wait_procs(procs_list, timeout=3, callback=on_terminate)
if len(gone)>0:
break
You can use os.wait() for this. You just have to call it in a loop until it tells you one of the processes you care about has exited.
import subprocess
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
# wait for the first process to finish
watched_pids = set(proc.pid for proc in (a, b))
while True:
pid, _ = os.wait()
if pid in watched_pids:
break
However, a hidden side effect of os.wait() is that you lose the process’s exit code. It will just be None after os.wait() finishes, and if you later call proc.wait(), proc.poll(), or proc.communicate() they will fail to find the return code and default to 0. It is possible to set it yourself, but it’s kind of hacky.
def wait_and_handle_exitstatus(all_procs):
pid, status = os.wait()
for proc in all_procs:
if proc.pid == pid:
# We need to set the process's exit status now, or we
# won't be able to retrieve it later and it will be
# assumed to be 0.
# This is a kind of hacky solution, but this function has existed
# ever since subprocess was first included in the stdlib and is
# still there in 3.10+, so it *should* be pretty stable.
proc._handle_exitstatus(status)
return pid, status
You can then use the first code block, only replace os.wait() with wait_and_handle_exitstatus(ALL_PROCS). You do, however, have to pass wait_and_handle_exitstatus a list of all subprocesses (Popen objects) that might be running and you might care about the return code of, so it can find the process and set its exit code.
Using asyncio.wait or asyncio.as_completed:
import asyncio
async def example():
p1 = await asyncio.create_subprocess_exec("sleep", "1")
p2 = await asyncio.create_subprocess_exec("sleep", "2")
p1_run = asyncio.create_task(p1.wait())
p2_run = asyncio.create_task(p2.wait())
pending = [p1_run, p2_run]
while pending:
done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
if p1_run in done:
print("p1 finished, with status: ", p1.returncode)
if p2_run in done:
print("p2 finished, with status: ", p2.returncode)
asyncio.get_event_loop().run_until_complete(example())
To avoid having to repeat which one of p1 and p2 that was done, you usually end up with a more complex mapping of the px_run to px.
To avoid that, another option is to wrap the task in something like wait_and_return_original below, next example also uses a more convenient asyncio.as_completed
async def wait_and_return_original(proc: asyncio.subprocess):
await proc.wait()
return proc
async def example2():
p1 = await asyncio.create_subprocess_exec("sleep", "1")
p2 = await asyncio.create_subprocess_exec("sleep", "2")
for p in asyncio.as_completed([wait_and_return_original(p) for p in [p1, p2]]):
p_completed = await p # NOTE: for-loop iteration variable doesn't decide which task is first completed until here!
if p_completed is p1:
print("p1 finished, with status: ", p1.returncode)
if p_completed is p2:
print("p2 finished, with status: ", p2.returncode)
asyncio.get_event_loop().run_until_complete(example2())
I have a list of subprocess‘ processes. I do not communicate with them and just wait.
I want to wait for the first process to finish (this solution works):
import subprocess
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
# wait for the first process to finish
while True:
over = False
for child in {a, b}:
try:
rst = child.wait(timeout=5)
except subprocess.TimeoutExpired:
continue # this subprocess is still running
if rst is not None: # subprocess is no more running
over = True
break # If either subprocess exits, so do we.
if over:
break
I don’t want use os.wait(), cause it could return from another subprocess not part of the list I’m waiting for.
A nice and elegant solution would probably be with an epoll or select and without any loop.
There are pretty much 2 ways to do this, if you want the commands to be blocking and halt your program until it finishes, use subprocess.call
a = subprocess.call('...')
b = subprocess.call('...')
I don’t think this is what you want though.
If you do not want them to halt your entire program and only need to check if one of them is finished before calling another, you should use .communicate
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
....
for child in {a, b}:
try:
result, err = child.communicate(timeout=5)
.communicate is pretty much the most elegant, simple and recommended solution
If you don’t need to get output from the processes, Popen.poll() seems to be the simplest way to check whether they are done. The while True loop below is purely for demonstration purposes: you can decide how to do this in your larger program (eg, do the checking in a separate thread, do the checking in between the other work of the program, etc).
from subprocess import Popen
from time import sleep
ps = [
Popen(['sleep', t])
for t in ('3', '5', '2')
]
while True:
exit_codes = [p.poll() for p in ps]
print(exit_codes)
if any(ec is not None for ec in exit_codes):
break
else:
sleep(1)
Demo output:
[None, None, None]
[None, None, None]
[None, None, 0]
Here’s a solution using psutil – which is aimed exactly at this use-case:
import subprocess
import psutil
a = subprocess.Popen(['/bin/sleep', "2"])
b = subprocess.Popen(['/bin/sleep', "4"])
procs_list = [psutil.Process(a.pid), psutil.Process(b.pid)]
def on_terminate(proc):
print("process {} terminated".format(proc))
# waits for multiple processes to terminate
gone, alive = psutil.wait_procs(procs_list, timeout=3, callback=on_terminate)
Or, if you’d like to have a loop waiting for one of the process to be done:
while True:
gone, alive = psutil.wait_procs(procs_list, timeout=3, callback=on_terminate)
if len(gone)>0:
break
You can use os.wait() for this. You just have to call it in a loop until it tells you one of the processes you care about has exited.
import subprocess
a = subprocess.Popen(['...'])
b = subprocess.Popen(['...'])
# wait for the first process to finish
watched_pids = set(proc.pid for proc in (a, b))
while True:
pid, _ = os.wait()
if pid in watched_pids:
break
However, a hidden side effect of os.wait() is that you lose the process’s exit code. It will just be None after os.wait() finishes, and if you later call proc.wait(), proc.poll(), or proc.communicate() they will fail to find the return code and default to 0. It is possible to set it yourself, but it’s kind of hacky.
def wait_and_handle_exitstatus(all_procs):
pid, status = os.wait()
for proc in all_procs:
if proc.pid == pid:
# We need to set the process's exit status now, or we
# won't be able to retrieve it later and it will be
# assumed to be 0.
# This is a kind of hacky solution, but this function has existed
# ever since subprocess was first included in the stdlib and is
# still there in 3.10+, so it *should* be pretty stable.
proc._handle_exitstatus(status)
return pid, status
You can then use the first code block, only replace os.wait() with wait_and_handle_exitstatus(ALL_PROCS). You do, however, have to pass wait_and_handle_exitstatus a list of all subprocesses (Popen objects) that might be running and you might care about the return code of, so it can find the process and set its exit code.
Using asyncio.wait or asyncio.as_completed:
import asyncio
async def example():
p1 = await asyncio.create_subprocess_exec("sleep", "1")
p2 = await asyncio.create_subprocess_exec("sleep", "2")
p1_run = asyncio.create_task(p1.wait())
p2_run = asyncio.create_task(p2.wait())
pending = [p1_run, p2_run]
while pending:
done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
if p1_run in done:
print("p1 finished, with status: ", p1.returncode)
if p2_run in done:
print("p2 finished, with status: ", p2.returncode)
asyncio.get_event_loop().run_until_complete(example())
To avoid having to repeat which one of p1 and p2 that was done, you usually end up with a more complex mapping of the px_run to px.
To avoid that, another option is to wrap the task in something like wait_and_return_original below, next example also uses a more convenient asyncio.as_completed
async def wait_and_return_original(proc: asyncio.subprocess):
await proc.wait()
return proc
async def example2():
p1 = await asyncio.create_subprocess_exec("sleep", "1")
p2 = await asyncio.create_subprocess_exec("sleep", "2")
for p in asyncio.as_completed([wait_and_return_original(p) for p in [p1, p2]]):
p_completed = await p # NOTE: for-loop iteration variable doesn't decide which task is first completed until here!
if p_completed is p1:
print("p1 finished, with status: ", p1.returncode)
if p_completed is p2:
print("p2 finished, with status: ", p2.returncode)
asyncio.get_event_loop().run_until_complete(example2())