why does python.subprocess hang after proc.communicate()?
Question:
I’ve got an interactive program called my_own_exe. First, it prints out alive, then you input Sn and then it prints out alive again. Finally you input Ln. It does some processing and exits.
However, when I call it from the following python script, the program seemed to hang after printing out the first ‘alive’.
Can anyone here tell me why this is happening?
// after reading the follow ups (thank you guys), i modified the code as following:
import subprocess
import time
base_command = "./AO_FelixStrategy_UnitTest --bats 31441 --chix 12467 --enxutp 31884 --turq 26372 --symbol SOGN --target_date " + '2009-Oct-16'
print base_command
proc2 = subprocess.Popen(base_command, shell=True , stdin=subprocess.PIPE,)
time.sleep(2);
print "aliv"
proc2.communicate('Sn')
print "alive"
time.sleep(6)
print "alive"
print proc2.communicate('Ln')
time.sleep(6)
the program now goes well with the first input ‘Sn’, but then stopped, and I the second ‘Ln’ is kinda ignored.
Can anyone give me an idea why it’s like this?
Answers:
From the docs for communicate:
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So after communicate() runs, the process has been terminated.
If you want to write and read without waiting for the process to stop:
-
Don’t ever use shell=True – it needlessy invokes a shell to in turn call your program, so there will be another process between you and your program. That has lots of unpleasant side-effects. The default is shell=False so you should stick with that.
Change your Popen line to:
p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
"--bats", "31441", "--chix", "12467",
"--enxutp", "31884", "--turq", "26372",
"--symbol", "SOGN", "--target_date", '2009-Oct-16'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
-
Use p.stdin.write to write to the process. Use p.stdout.read to read from it.
- Calling
p.stdout.read if there’s nothing to read will block. Calling p.stdin.write if the write buffer is full will block. So you have to make sure you have something to read/write – you do that on unix OS by using select. On windows you unfortunately must resort to threads. At least that is what Popen.communicate does internally.
- If you didn’t write
AO_FelixStrategy_UnitTest then you have possible additional problems:
- It could be reading from somewhere else, not standard input. Some programs read directly from the terminal, others use some OS API to read. That means data written to stdin won’t go to the program. This is often true for password prompts.
- Remember that you have to account for
AO_FelixStrategy_UnitTest buffers. By default standard C PIPE communication is buffered so you may not see any output until after you’ve closed the input side (by doing p.stdin.close(). Unless AO_FelixStrategy_UnitTest flushes the output periodically.
Here’s some example code, based on what you describe. It could work depending on how AO_FelixStrategy_UnitTest was developed:
p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
"--bats", "31441", "--chix", "12467",
"--enxutp", "31884", "--turq", "26372",
"--symbol", "SOGN", "--target_date", '2009-Oct-16'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
output = p.communicate('SnLn')[0]
print output
communicate() reads data from stdout and stderr until end-of-file is reached. – It waits until your program quits.
comunicate will only run once and then will close the pipe, therefore if you want to send several commands to it you need to send one after the other in the same string.
Here is an example that worked for me after some investigation, trying threads, subprocess32, stdin.write, stdout.read etc etc. This information is not in the official python reference information for communicate: https://docs.python.org/2/library/subprocess.html
The only place where I found this out was here:
Python subprocess communicate kills my process
Here is the code, it is simple, no threads, no subprocess32, works on linux and windows. Yes you have to know how many times to send commands to the other process but in general you do know that. On top of this you can add threads, cwd, shell=True or anything else you may want but this is the simplest case:
def do_commands(self, cmd, parms):
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE )
# wait for the process to terminate
out, err = process.communicate(cmd_parms)
errcode = process.returncode
return errcode, out, err
So for example if you want to send multiple carriage returns (n) to the application being called and the a param in the middle (interactively mind you) you would call it something like this:
cmd_parms = "nnnnnparmnn"
errcode, out, err = do_commands(command, cmd_parms)
Hope it helps.
Although the answers above say that the process.communicate() is safe after the process has been terminated (the process has quit), there is a caveat.
-
If the process has spawned any child processes, they seem to be blocking process.communicate() as long as child processes are still alive after the calling kill() for the parent. It seems that process.wait() may return a return code for the process (the process has exited), but process.communicate() will still hang if called.
-
This is because of how UNIX-based operating systems work (Linux/macOS)
-
Events leading to this situation can subtly happen if multiprocessing.Manager Python class is used from the standard library. Manager will spawn a child process behind the scenes. This is not visible to the developer, so this may came as a surprise.
-
Because the child process Manager has spawned does not have Process.daemon flag set, this spawned process will block the terminate() or kill()for the whole process group, causing process.communicate() not to work even after the process has seemingly terminated
-
In this stage, the process is in a "zombie" state, not dead but not alive either
For the reference, I did not find a straightforward solution on how to safely use Manager if you want to use process.communicate().
I’ve got an interactive program called my_own_exe. First, it prints out alive, then you input Sn and then it prints out alive again. Finally you input Ln. It does some processing and exits.
However, when I call it from the following python script, the program seemed to hang after printing out the first ‘alive’.
Can anyone here tell me why this is happening?
// after reading the follow ups (thank you guys), i modified the code as following:
import subprocess
import time
base_command = "./AO_FelixStrategy_UnitTest --bats 31441 --chix 12467 --enxutp 31884 --turq 26372 --symbol SOGN --target_date " + '2009-Oct-16'
print base_command
proc2 = subprocess.Popen(base_command, shell=True , stdin=subprocess.PIPE,)
time.sleep(2);
print "aliv"
proc2.communicate('Sn')
print "alive"
time.sleep(6)
print "alive"
print proc2.communicate('Ln')
time.sleep(6)
the program now goes well with the first input ‘Sn’, but then stopped, and I the second ‘Ln’ is kinda ignored.
Can anyone give me an idea why it’s like this?
From the docs for communicate:
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
So after communicate() runs, the process has been terminated.
If you want to write and read without waiting for the process to stop:
-
Don’t ever use
shell=True– it needlessy invokes a shell to in turn call your program, so there will be another process between you and your program. That has lots of unpleasant side-effects. The default isshell=Falseso you should stick with that.
Change yourPopenline to:p = subprocess.Popen(["./AO_FelixStrategy_UnitTest", "--bats", "31441", "--chix", "12467", "--enxutp", "31884", "--turq", "26372", "--symbol", "SOGN", "--target_date", '2009-Oct-16'], stdin=subprocess.PIPE, stdout=subprocess.PIPE) -
Use
p.stdin.writeto write to the process. Usep.stdout.readto read from it. - Calling
p.stdout.readif there’s nothing to read will block. Callingp.stdin.writeif the write buffer is full will block. So you have to make sure you have something to read/write – you do that on unix OS by usingselect. On windows you unfortunately must resort to threads. At least that is whatPopen.communicatedoes internally. - If you didn’t write
AO_FelixStrategy_UnitTestthen you have possible additional problems:- It could be reading from somewhere else, not standard input. Some programs read directly from the terminal, others use some OS API to read. That means data written to stdin won’t go to the program. This is often true for password prompts.
- Remember that you have to account for
AO_FelixStrategy_UnitTestbuffers. By default standard C PIPE communication is buffered so you may not see any output until after you’ve closed the input side (by doingp.stdin.close(). UnlessAO_FelixStrategy_UnitTestflushes the output periodically.
Here’s some example code, based on what you describe. It could work depending on how AO_FelixStrategy_UnitTest was developed:
p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
"--bats", "31441", "--chix", "12467",
"--enxutp", "31884", "--turq", "26372",
"--symbol", "SOGN", "--target_date", '2009-Oct-16'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
output = p.communicate('SnLn')[0]
print output
communicate() reads data from stdout and stderr until end-of-file is reached. – It waits until your program quits.
comunicate will only run once and then will close the pipe, therefore if you want to send several commands to it you need to send one after the other in the same string.
Here is an example that worked for me after some investigation, trying threads, subprocess32, stdin.write, stdout.read etc etc. This information is not in the official python reference information for communicate: https://docs.python.org/2/library/subprocess.html
The only place where I found this out was here:
Python subprocess communicate kills my process
Here is the code, it is simple, no threads, no subprocess32, works on linux and windows. Yes you have to know how many times to send commands to the other process but in general you do know that. On top of this you can add threads, cwd, shell=True or anything else you may want but this is the simplest case:
def do_commands(self, cmd, parms):
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE )
# wait for the process to terminate
out, err = process.communicate(cmd_parms)
errcode = process.returncode
return errcode, out, err
So for example if you want to send multiple carriage returns (n) to the application being called and the a param in the middle (interactively mind you) you would call it something like this:
cmd_parms = "nnnnnparmnn"
errcode, out, err = do_commands(command, cmd_parms)
Hope it helps.
Although the answers above say that the process.communicate() is safe after the process has been terminated (the process has quit), there is a caveat.
-
If the process has spawned any child processes, they seem to be blocking
process.communicate()as long as child processes are still alive after the callingkill()for the parent. It seems thatprocess.wait()may return a return code for the process (the process has exited), butprocess.communicate()will still hang if called. -
This is because of how UNIX-based operating systems work (Linux/macOS)
-
Events leading to this situation can subtly happen if
multiprocessing.ManagerPython class is used from the standard library.Managerwill spawn a child process behind the scenes. This is not visible to the developer, so this may came as a surprise. -
Because the child process
Managerhas spawned does not have Process.daemon flag set, this spawned process will block theterminate()orkill()for the whole process group, causingprocess.communicate()not to work even after the process has seemingly terminated -
In this stage, the process is in a "zombie" state, not dead but not alive either
For the reference, I did not find a straightforward solution on how to safely use Manager if you want to use process.communicate().