What's the correct way to use win32file.ReadFile to get the output from a pipe?
Question:
I’m using the pywin32 extensions to access the win32 API under Python. I’m new at doing Windows programming in Python — I’m a POSIX guy — so I may be doing things in a bone-headed manner.
I’m trying to use the win32file.ReadFile function properly, and I’m having some trouble interpreting the possible result codes.
I’m calling the function like this:
result, data = win32file.ReadFile(child_stdout_r, 4096, None)
I’m reading the output from a child process that I launch. I get good data, but I’m concerned that there may be more data in the pipe than 4096 characters. (And I’d rather do this right, instead of just picking an arbitrarily large buffer size.)
In the case where there’s more than 4096 characters to read, I would need to run win32file.ReadFile multiple times until I exhaust the pipe. To find out whether I need to run ReadFile multiple times, I need to interpret the result code.
The ActiveState docs say that:
The result is a tuple of (hr, string/PyOVERLAPPEDReadBuffer), where hr may be 0, ERROR_MORE_DATA or ERROR_IO_PENDING.
Since I’m setting the overlapped value to None in the function call, I think I don’t need to worry about any PyOVERLAPPEDReadBuffer stuff. (And since I’m getting valid data, I think I’m right.)
I have two problems with the hr result variable:
- I can’t find the values of the constants ERROR_MORE_DATA or ERROR_IO_PENDING anywhere.
- The ActiveState docs seem to imply that 0 is success and the constants (whatever they are) indicate failure. The Microsoft docs state that 0 indicates failure, non-zero indicates success, and you need to run GetLastError to find out more.
What’s the correct way to do this?
EDITED TO ADD: I’m not using subprocess because I need to add the child process to a job object I create. The goal is to have all child processes die immediately if the parent process dies. By adding the child process to the job object, the child process will be terminated when the last handle to the job object is closed. The handle, held by the parent, will be closed when the parent exits. All of this, as far as I can tell, precludes me from using subprocess.
Answers:
Consider using subprocess to launch the process. It will give you a set of file-like objects that you can use to talk with the other app.
The .terminate() method of the Popen object will allow you to terminate the process if you are running 2.6+.
For error codes, try winerror.ERROR_MORE_DATA and winerror.ERROR_IO_PENDING
My interpretation of the ActiveState docs is the same as yours. It sounds like the wrapper works slightly differently than the native API.
I haven’t actually tried this.
note that ReadFile is defined as:
(int, string) = ReadFile(hFile, buffer/bufSize , overlapped)
where…
hFile = PyHANDLE
which is any windows handle (can be file, process, thread…)
buffer/bufSize = PyOVERLAPPEDReadBuffer
which, according to documentation automatically allocates contents of hFile regardless if it overlaps or not.
overlapped=None [=PyOVERLAPPED]
you can allocate an additional object to take any extra data, beyond the overlapped (buffer/bufSize) if you wish, but by default this is NULL.
So – you can basically call ReadFile like:
ReadFile(child_stdout_r, 0, None)
and the object you assign it to will contain the full contents of the file handle.
I’m using the pywin32 extensions to access the win32 API under Python. I’m new at doing Windows programming in Python — I’m a POSIX guy — so I may be doing things in a bone-headed manner.
I’m trying to use the win32file.ReadFile function properly, and I’m having some trouble interpreting the possible result codes.
I’m calling the function like this:
result, data = win32file.ReadFile(child_stdout_r, 4096, None)
I’m reading the output from a child process that I launch. I get good data, but I’m concerned that there may be more data in the pipe than 4096 characters. (And I’d rather do this right, instead of just picking an arbitrarily large buffer size.)
In the case where there’s more than 4096 characters to read, I would need to run win32file.ReadFile multiple times until I exhaust the pipe. To find out whether I need to run ReadFile multiple times, I need to interpret the result code.
The ActiveState docs say that:
The result is a tuple of (hr, string/PyOVERLAPPEDReadBuffer), where hr may be 0, ERROR_MORE_DATA or ERROR_IO_PENDING.
Since I’m setting the overlapped value to None in the function call, I think I don’t need to worry about any PyOVERLAPPEDReadBuffer stuff. (And since I’m getting valid data, I think I’m right.)
I have two problems with the hr result variable:
- I can’t find the values of the constants ERROR_MORE_DATA or ERROR_IO_PENDING anywhere.
- The ActiveState docs seem to imply that 0 is success and the constants (whatever they are) indicate failure. The Microsoft docs state that 0 indicates failure, non-zero indicates success, and you need to run GetLastError to find out more.
What’s the correct way to do this?
EDITED TO ADD: I’m not using subprocess because I need to add the child process to a job object I create. The goal is to have all child processes die immediately if the parent process dies. By adding the child process to the job object, the child process will be terminated when the last handle to the job object is closed. The handle, held by the parent, will be closed when the parent exits. All of this, as far as I can tell, precludes me from using subprocess.
Consider using subprocess to launch the process. It will give you a set of file-like objects that you can use to talk with the other app.
The .terminate() method of the Popen object will allow you to terminate the process if you are running 2.6+.
For error codes, try winerror.ERROR_MORE_DATA and winerror.ERROR_IO_PENDING
My interpretation of the ActiveState docs is the same as yours. It sounds like the wrapper works slightly differently than the native API.
I haven’t actually tried this.
note that ReadFile is defined as:
(int, string) = ReadFile(hFile, buffer/bufSize , overlapped)
where…
hFile = PyHANDLE
which is any windows handle (can be file, process, thread…)
buffer/bufSize = PyOVERLAPPEDReadBuffer
which, according to documentation automatically allocates contents of hFile regardless if it overlaps or not.
overlapped=None [=PyOVERLAPPED]
you can allocate an additional object to take any extra data, beyond the overlapped (buffer/bufSize) if you wish, but by default this is NULL.
So – you can basically call ReadFile like:
ReadFile(child_stdout_r, 0, None)
and the object you assign it to will contain the full contents of the file handle.