Python subprocess .check_call vs .check_output
Question:
My python script (python 3.4.3) calls a bash script via subprocess:
import subprocess as sp
res = sp.check_output("bashscript", shell=True)
The bashscript contains the following line:
ssh -MNf somehost
which opens a shared master connection to some remote host to allow some subsequent operations.
When executing the python script, it will prompt for password for the ssh line but then it blocks after the password is entered and never returns. When I ctrl-C to terminate the script, I see that the connection was properly established (so ssh line was successfully executed).
I don’t have this blocking problem when using check_call instead of check_output, but check_call does not retrieve stdout. I’d like to understand what exactly is causing the blocking behavior for check_output, probably related to some subtlety with ssh -MNf.
Answers:
check_call() returns as soon as /bin/sh process exits without waiting for descendant processes (assuming shell=True as in your case).
check_output() waits until all output is read. If ssh inherits the pipe then check_output() will wait until it exits (until it closes its inherited pipe ends).
check_call() code example:
#!/usr/bin/env python
import subprocess
import sys
import time
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' &"
subprocess.check_call(cmd, shell=True)
assert (time.time() - start) < 1
The output is not read; check_call() returns immediately without waiting for the grandchild background python process.
check_call() is just Popen().wait(). Popen() starts the external process and returns immediately without waiting for it to exit. .wait() collects the exit status for the process — it doesn’t wait for other (grandchildren) processes.
If the output is read (it is redirected and the grandchild python
process inherits the stdout pipe):
start = time.time()
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) > 2
then it waits until the background python process that inherited the pipe exits.
check_output() calls Popen().communicate(), to get the output. .communicate() calls .wait() internally i.e., check_output() also waits for the shell to exit and check_output() waits for EOF.
If the grandchild doesn’t inherit the pipe then check_output() doesn’t wait for it:
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' >/dev/null &"
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) < 1
Grandchild’s output is redirected to /dev/null i.e., it doesn’t inherit the parent’s pipe and therefore check_output() may exit without waiting for it.
Note: & at the end which puts the grandchild python process into background. It won’t work on Windows where shell=True starts cmd.exe by default.
My python script (python 3.4.3) calls a bash script via subprocess:
import subprocess as sp
res = sp.check_output("bashscript", shell=True)
The bashscript contains the following line:
ssh -MNf somehost
which opens a shared master connection to some remote host to allow some subsequent operations.
When executing the python script, it will prompt for password for the ssh line but then it blocks after the password is entered and never returns. When I ctrl-C to terminate the script, I see that the connection was properly established (so ssh line was successfully executed).
I don’t have this blocking problem when using check_call instead of check_output, but check_call does not retrieve stdout. I’d like to understand what exactly is causing the blocking behavior for check_output, probably related to some subtlety with ssh -MNf.
check_call() returns as soon as /bin/sh process exits without waiting for descendant processes (assuming shell=True as in your case).
check_output() waits until all output is read. If ssh inherits the pipe then check_output() will wait until it exits (until it closes its inherited pipe ends).
check_call() code example:
#!/usr/bin/env python
import subprocess
import sys
import time
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' &"
subprocess.check_call(cmd, shell=True)
assert (time.time() - start) < 1
The output is not read; check_call() returns immediately without waiting for the grandchild background python process.
check_call() is just Popen().wait(). Popen() starts the external process and returns immediately without waiting for it to exit. .wait() collects the exit status for the process — it doesn’t wait for other (grandchildren) processes.
If the output is read (it is redirected and the grandchild python
process inherits the stdout pipe):
start = time.time()
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) > 2
then it waits until the background python process that inherited the pipe exits.
check_output() calls Popen().communicate(), to get the output. .communicate() calls .wait() internally i.e., check_output() also waits for the shell to exit and check_output() waits for EOF.
If the grandchild doesn’t inherit the pipe then check_output() doesn’t wait for it:
start = time.time()
cmd = sys.executable + " -c 'import time; time.sleep(2)' >/dev/null &"
subprocess.check_output(cmd, shell=True)
assert (time.time() - start) < 1
Grandchild’s output is redirected to /dev/null i.e., it doesn’t inherit the parent’s pipe and therefore check_output() may exit without waiting for it.
Note: & at the end which puts the grandchild python process into background. It won’t work on Windows where shell=True starts cmd.exe by default.