How to prevent BrokenPipeError when doing a flush in Python?
Question:
Question: Is there a way to use flush=True
for the print()
function without getting the BrokenPipeError
?
I have a script pipe.py
:
for i in range(4000):
print(i)
I call it like this from a Unix command line:
python3 pipe.py | head -n3000
And it returns:
0
1
2
So does this script:
import sys
for i in range(4000):
print(i)
sys.stdout.flush()
However, when I run this script and pipe it to head -n3000
:
for i in range(4000):
print(i, flush=True)
Then I get this error:
print(i, flush=True)
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored
I have also tried the solution below, but I still get the BrokenPipeError
:
import sys
for i in range(4000):
try:
print(i, flush=True)
except BrokenPipeError:
sys.exit()
Answers:
According to the Python documentation, this is thrown when:
trying to write on a pipe while the other end has been closed
This is due to the fact that the head utility reads from stdout
, then promptly closes it.
As you can see, it can be worked around by merely adding a sys.stdout.flush()
after every print()
. Note that this sometimes does not work in Python 3.
You can alternatively pipe it to awk
like this to get the same result as head -3
:
python3 0to3.py | awk 'NR >= 4 {exit} 1'
Hope this helped, good luck!
As you can see in the output that you had posted the last exception is raised in the destructor phase : that is why you have ignored
at the end
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored
A simple example to understand what’s up in that context is the follow:
>> class A():
... def __del__(self):
... raise Exception("It will be ignored!!!")
...
>>> a = A()
>>> del a
Exception Exception: Exception('It will be ignored!!!',) in <bound method A.__del__ of <__builtin__.A instance at 0x7ff1d5c06d88>> ignored
>>> a = A()
>>> import sys
>>> sys.stderr.close()
>>> del a
Every exception that is triggered while the object is destroyed will cause a standard error output that explain the exception occurred and ignored (that is because python will inform you that something could not be correctly handle in destroy phase). Anyway, that kind of exceptions cannot be cached and so you can just remove the calls that can generate it or close stderr
.
Come back to the question. That exception is not a real problem (as say it is ignored) but if you don’t want print it you must override the the function that can be called when the object will be destroyed or close stderr
as @SergeBallesta correctly suggested : in you case you can shutdown write
and flush
function and no exception will be triggered in destroy context
That is an example of how you can do it:
import sys
def _void_f(*args,**kwargs):
pass
for i in range(4000):
try:
print(i,flush=True)
except (BrokenPipeError, IOError):
sys.stdout.write = _void_f
sys.stdout.flush = _void_f
sys.exit()
The BrokenPipeError
is normal as said phantom because the reading process (head) terminates and closes its end of the pipe while the writing process (python) still tries to write.
Is is an abnormal condition, and the python scripts receives a BrokenPipeError
– more exactly, the Python interpreter receives a system SIGPIPE signal that it catches and raises the BrokenPipeError
to allow the script to process the error.
And you effectively can process the error, because in your last example, you only see a message saying that the exception was ignored – ok it is not true, but seems related to this open issue in Python : Python developpers think important to warn user of the abnormal condition.
What really happens is that AFAIK the python interpreter always signals this on stderr, even if you catch the exception. But you just have to close stderr before exiting to get rid of the message.
I slightly changed your script to :
- catch the error as you did in your last example
- catch either IOError (that I get in Python34 on Windows64) or BrokenPipeError (in Python 33 on FreeBSD 9.0) – and display a message for that
- display a custom Done message on stderr (stdout is closed due to the broken pipe)
- close stderr before exiting to get rid of the message
Here is the script I used :
import sys
try:
for i in range(4000):
print(i, flush=True)
except (BrokenPipeError, IOError):
print ('BrokenPipeError caught', file = sys.stderr)
print ('Done', file=sys.stderr)
sys.stderr.close()
and here the result of python3.3 pipe.py | head -10
:
0
1
2
3
4
5
6
7
8
9
BrokenPipeError caught
Done
If you do not want the extraneous messages just use :
import sys
try:
for i in range(4000):
print(i, flush=True)
except (BrokenPipeError, IOError):
pass
sys.stderr.close()
Ignore SIGPPIE temporarily
I’m not sure how bad an idea this is, but it works:
#!/usr/bin/env python3
import signal
import sys
sigpipe_old = signal.getsignal(signal.SIGPIPE)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for i in range(4000):
print(i, flush=True)
signal.signal(signal.SIGPIPE, sigpipe_old)
While others have covered the underlying issue in great detail there is a straightforward workaround:
python whatever.py | tail -n +1 | head -n3000
Explanation: tail
buffers until it’s STDIN is closed (python quits and closes its STDOUT). So only tail gets the SIGPIPE when head quits. The -n +1
is effectively a no-op, making tail output the “tail” starting at line 1, which is the entire buffer.
I’ve often wished there were a command-line option to suppress these signal handlers.
import signal
# Don't turn these signal into exceptions, just die.
signal.signal(signal.SIGINT, signal.SIG_DFL)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
Instead, the best we can do is uninstall the handlers as soon as possible as the Python script starts running.
A note on SIGPIPE
was added in Python 3.7 documentation, and it recommends to catch BrokenPipeError
this way:
import os
import sys
def main():
try:
# simulate large output (your code replaces this loop)
for x in range(10000):
print("y")
# flush output here to force SIGPIPE to be triggered
# while inside this try block.
sys.stdout.flush()
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
if __name__ == '__main__':
main()
Importantly, it says:
Do not set SIGPIPE
’s disposition to SIG_DFL
in order to avoid BrokenPipeError
. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.
Answer
import sys
for i in range(4000):
try:
print(i, flush=True)
except BrokenPipeError:
sys.stdout = None
Explanation
Even if you catch the BrokenPipeError exception, it will be thrown again by Python when your program exits and Python tries to flush stdout. By setting stdout to None, Python will not attempt to flush it.
Drawbacks
While Python routines, such as print()
, correctly check if stdout is None and will not fail, it is not uncommon to see programs that do not check. If your program attempts to use stdout.write()
, or similar, after setting stdout to None, then Python will throw an AttributeError.
Other answers (and why not)
No answer is shorter or simpler than sys.stdout = None
, but some of the common answers have significant problems.
/dev/null
The Python developers have their own suggested code for dealing with the BrokenPipeError.
import os
import sys
def main():
try:
# simulate large output (your code replaces this loop)
for x in range(10000):
print("y")
# flush output here to force SIGPIPE to be triggered
# while inside this try block.
sys.stdout.flush()
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
if __name__ == '__main__':
main()
While that is the canonical answer, it is rather grotesque in that it needlessly opens a new file descriptor to /dev/null just so that Python can flush it before it is closed.
Why not: For most people, it is pointless. This problem is caused by Python flushing a handle that we have already caught a BrokenPipeError on. We know it will fail, so the solution should be for Python to simply not flush that handle. To allocate a new file descriptor just to appease Python is silly.
Why (maybe): Redirecting stdout to /dev/null may actually be the right solution for some people whose programs, after receiving a BrokenPipeError, will continue manipulating stdout without checking it first. However, that is not the common case.
sys.stderr.close()
Some people have suggested closing stderr to hide the bogus BrokenPipe error message.
Why not: It also prevents any legitimate errors from being shown.
signal(SIGPIPE, SIG_DFL)
Another common answer is to use SIG_DFL
, the default signal handler, to cause the program to die whenever a SIGPIPE signal is received.
Why not: SIGPIPE can be sent for any file descriptor, not just stdout, so your entire program would suddenly and mysteriously die if, for example, it was writing to a network socket whose connection gets interrupted.
pipe.py | something | head
One non-python solution is to first pipe stdout to a program that will continue reading data from the Python program even when its own standard output is closed. For example, presuming you have the GNU version of tee
, this works:
pipe.py | tee -p /dev/null | head
Why not: The question was looking for an answer in Python. Also, it is suboptimal in that it will keep pipe.py running longer than it needs to, possible consuming significant resources.
Question: Is there a way to use flush=True
for the print()
function without getting the BrokenPipeError
?
I have a script pipe.py
:
for i in range(4000):
print(i)
I call it like this from a Unix command line:
python3 pipe.py | head -n3000
And it returns:
0
1
2
So does this script:
import sys
for i in range(4000):
print(i)
sys.stdout.flush()
However, when I run this script and pipe it to head -n3000
:
for i in range(4000):
print(i, flush=True)
Then I get this error:
print(i, flush=True)
BrokenPipeError: [Errno 32] Broken pipe
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored
I have also tried the solution below, but I still get the BrokenPipeError
:
import sys
for i in range(4000):
try:
print(i, flush=True)
except BrokenPipeError:
sys.exit()
According to the Python documentation, this is thrown when:
trying to write on a pipe while the other end has been closed
This is due to the fact that the head utility reads from stdout
, then promptly closes it.
As you can see, it can be worked around by merely adding a sys.stdout.flush()
after every print()
. Note that this sometimes does not work in Python 3.
You can alternatively pipe it to awk
like this to get the same result as head -3
:
python3 0to3.py | awk 'NR >= 4 {exit} 1'
Hope this helped, good luck!
As you can see in the output that you had posted the last exception is raised in the destructor phase : that is why you have ignored
at the end
Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored
A simple example to understand what’s up in that context is the follow:
>> class A():
... def __del__(self):
... raise Exception("It will be ignored!!!")
...
>>> a = A()
>>> del a
Exception Exception: Exception('It will be ignored!!!',) in <bound method A.__del__ of <__builtin__.A instance at 0x7ff1d5c06d88>> ignored
>>> a = A()
>>> import sys
>>> sys.stderr.close()
>>> del a
Every exception that is triggered while the object is destroyed will cause a standard error output that explain the exception occurred and ignored (that is because python will inform you that something could not be correctly handle in destroy phase). Anyway, that kind of exceptions cannot be cached and so you can just remove the calls that can generate it or close stderr
.
Come back to the question. That exception is not a real problem (as say it is ignored) but if you don’t want print it you must override the the function that can be called when the object will be destroyed or close stderr
as @SergeBallesta correctly suggested : in you case you can shutdown write
and flush
function and no exception will be triggered in destroy context
That is an example of how you can do it:
import sys
def _void_f(*args,**kwargs):
pass
for i in range(4000):
try:
print(i,flush=True)
except (BrokenPipeError, IOError):
sys.stdout.write = _void_f
sys.stdout.flush = _void_f
sys.exit()
The BrokenPipeError
is normal as said phantom because the reading process (head) terminates and closes its end of the pipe while the writing process (python) still tries to write.
Is is an abnormal condition, and the python scripts receives a BrokenPipeError
– more exactly, the Python interpreter receives a system SIGPIPE signal that it catches and raises the BrokenPipeError
to allow the script to process the error.
And you effectively can process the error, because in your last example, you only see a message saying that the exception was ignored – ok it is not true, but seems related to this open issue in Python : Python developpers think important to warn user of the abnormal condition.
What really happens is that AFAIK the python interpreter always signals this on stderr, even if you catch the exception. But you just have to close stderr before exiting to get rid of the message.
I slightly changed your script to :
- catch the error as you did in your last example
- catch either IOError (that I get in Python34 on Windows64) or BrokenPipeError (in Python 33 on FreeBSD 9.0) – and display a message for that
- display a custom Done message on stderr (stdout is closed due to the broken pipe)
- close stderr before exiting to get rid of the message
Here is the script I used :
import sys
try:
for i in range(4000):
print(i, flush=True)
except (BrokenPipeError, IOError):
print ('BrokenPipeError caught', file = sys.stderr)
print ('Done', file=sys.stderr)
sys.stderr.close()
and here the result of python3.3 pipe.py | head -10
:
0
1
2
3
4
5
6
7
8
9
BrokenPipeError caught
Done
If you do not want the extraneous messages just use :
import sys
try:
for i in range(4000):
print(i, flush=True)
except (BrokenPipeError, IOError):
pass
sys.stderr.close()
Ignore SIGPPIE temporarily
I’m not sure how bad an idea this is, but it works:
#!/usr/bin/env python3
import signal
import sys
sigpipe_old = signal.getsignal(signal.SIGPIPE)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for i in range(4000):
print(i, flush=True)
signal.signal(signal.SIGPIPE, sigpipe_old)
While others have covered the underlying issue in great detail there is a straightforward workaround:
python whatever.py | tail -n +1 | head -n3000
Explanation: tail
buffers until it’s STDIN is closed (python quits and closes its STDOUT). So only tail gets the SIGPIPE when head quits. The -n +1
is effectively a no-op, making tail output the “tail” starting at line 1, which is the entire buffer.
I’ve often wished there were a command-line option to suppress these signal handlers.
import signal
# Don't turn these signal into exceptions, just die.
signal.signal(signal.SIGINT, signal.SIG_DFL)
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
Instead, the best we can do is uninstall the handlers as soon as possible as the Python script starts running.
A note on SIGPIPE
was added in Python 3.7 documentation, and it recommends to catch BrokenPipeError
this way:
import os
import sys
def main():
try:
# simulate large output (your code replaces this loop)
for x in range(10000):
print("y")
# flush output here to force SIGPIPE to be triggered
# while inside this try block.
sys.stdout.flush()
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
if __name__ == '__main__':
main()
Importantly, it says:
Do not set
SIGPIPE
’s disposition toSIG_DFL
in order to avoidBrokenPipeError
. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it.
Answer
import sys
for i in range(4000):
try:
print(i, flush=True)
except BrokenPipeError:
sys.stdout = None
Explanation
Even if you catch the BrokenPipeError exception, it will be thrown again by Python when your program exits and Python tries to flush stdout. By setting stdout to None, Python will not attempt to flush it.
Drawbacks
While Python routines, such as print()
, correctly check if stdout is None and will not fail, it is not uncommon to see programs that do not check. If your program attempts to use stdout.write()
, or similar, after setting stdout to None, then Python will throw an AttributeError.
Other answers (and why not)
No answer is shorter or simpler than sys.stdout = None
, but some of the common answers have significant problems.
/dev/null
The Python developers have their own suggested code for dealing with the BrokenPipeError.
import os
import sys
def main():
try:
# simulate large output (your code replaces this loop)
for x in range(10000):
print("y")
# flush output here to force SIGPIPE to be triggered
# while inside this try block.
sys.stdout.flush()
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
# to devnull to avoid another BrokenPipeError at shutdown
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
sys.exit(1) # Python exits with error code 1 on EPIPE
if __name__ == '__main__':
main()
While that is the canonical answer, it is rather grotesque in that it needlessly opens a new file descriptor to /dev/null just so that Python can flush it before it is closed.
Why not: For most people, it is pointless. This problem is caused by Python flushing a handle that we have already caught a BrokenPipeError on. We know it will fail, so the solution should be for Python to simply not flush that handle. To allocate a new file descriptor just to appease Python is silly.
Why (maybe): Redirecting stdout to /dev/null may actually be the right solution for some people whose programs, after receiving a BrokenPipeError, will continue manipulating stdout without checking it first. However, that is not the common case.
sys.stderr.close()
Some people have suggested closing stderr to hide the bogus BrokenPipe error message.
Why not: It also prevents any legitimate errors from being shown.
signal(SIGPIPE, SIG_DFL)
Another common answer is to use SIG_DFL
, the default signal handler, to cause the program to die whenever a SIGPIPE signal is received.
Why not: SIGPIPE can be sent for any file descriptor, not just stdout, so your entire program would suddenly and mysteriously die if, for example, it was writing to a network socket whose connection gets interrupted.
pipe.py | something | head
One non-python solution is to first pipe stdout to a program that will continue reading data from the Python program even when its own standard output is closed. For example, presuming you have the GNU version of tee
, this works:
pipe.py | tee -p /dev/null | head
Why not: The question was looking for an answer in Python. Also, it is suboptimal in that it will keep pipe.py running longer than it needs to, possible consuming significant resources.