Communicate multiple times with a process without breaking the pipe?
Question:
It’s not the first time I’m having this problem, and it’s really bugging me.
Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached
proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099n")[0]
The problem here is that the second time, Python isn’t happy. Indeed, he decided to close the file after the first communicate:
Traceback (most recent call last):
File "a.py", line 30, in <module>
print proc.communicate("select theta,zeta,result from experiment_2099n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
Are multiple communications allowed?
Answers:
I’ve had this problem before, and as far as I could ever figure, you couldn’t do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).
I think you misunderstand communicate…
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
communicate sends a string to the other process and then waits on it to finish… (Like you said waits for the EOF listening to the stdout & stderror)
What you should do instead is:
proc.stdin.write('message')
# ...figure out how long or why you need to wait...
proc.stdin.write('message2')
(and if you need to get the stdout or stderr you’d use proc.stdout or proc.stderr)
You can use:
proc.stdin.write('input')
if proc.stdout.closed:
print(proc.stdout)
You can do this simply with single call of communicate():
query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])
The key-point here is that you only write once to stdin, and n serve as EOL.
your psql subprocess reads from stdin until n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.
It’s not the first time I’m having this problem, and it’s really bugging me.
Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached
proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099n")[0]
The problem here is that the second time, Python isn’t happy. Indeed, he decided to close the file after the first communicate:
Traceback (most recent call last):
File "a.py", line 30, in <module>
print proc.communicate("select theta,zeta,result from experiment_2099n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
Are multiple communications allowed?
I’ve had this problem before, and as far as I could ever figure, you couldn’t do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).
I think you misunderstand communicate…
http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
communicate sends a string to the other process and then waits on it to finish… (Like you said waits for the EOF listening to the stdout & stderror)
What you should do instead is:
proc.stdin.write('message')
# ...figure out how long or why you need to wait...
proc.stdin.write('message2')
(and if you need to get the stdout or stderr you’d use proc.stdout or proc.stderr)
You can use:
proc.stdin.write('input')
if proc.stdout.closed:
print(proc.stdout)
You can do this simply with single call of communicate():
query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])
The key-point here is that you only write once to stdin, and n serve as EOL.
your psql subprocess reads from stdin until n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.