Suppress stdout / stderr print from Python functions
Question:
I have a Python script that is using some closed-box Python functions (i.e. I can’t edit these functions) provided by my employer. When I call these functions, they are printing output to my linux terminal that I would like to suppress. I’ve tried redirecting stdout / stderr via;
orig_out = sys.stdout
sys.stdout = StringIO()
rogue_function()
sys.stdout = orig_out
but this fails to catch the output. I think the functions I’m calling via-Python (rogue_function() from above) are really wrappers for compiled C-code, which are actually doing the printing.
Does anyone know of a way I can do a “deep-capture” of any print handed to stdout / stderr by a function (and any sub-functions that function calls)?
UPDATE:
I ended up taking the method outlined in the selected answer below and writing a context manager to supress stdout and stderr:
# Define a context manager to suppress stdout and stderr.
class suppress_stdout_stderr(object):
'''
A context manager for doing a "deep suppression" of stdout and stderr in
Python, i.e. will suppress all print, even if the print originates in a
compiled C/Fortran sub-function.
This will not suppress raised exceptions, since exceptions are printed
to stderr just before a script exits, and after the context manager has
exited (at least, I think that is why it lets exceptions through).
'''
def __init__(self):
# Open a pair of null files
self.null_fds = [os.open(os.devnull,os.O_RDWR) for x in range(2)]
# Save the actual stdout (1) and stderr (2) file descriptors.
self.save_fds = [os.dup(1), os.dup(2)]
def __enter__(self):
# Assign the null pointers to stdout and stderr.
os.dup2(self.null_fds[0],1)
os.dup2(self.null_fds[1],2)
def __exit__(self, *_):
# Re-assign the real stdout/stderr back to (1) and (2)
os.dup2(self.save_fds[0],1)
os.dup2(self.save_fds[1],2)
# Close all file descriptors
for fd in self.null_fds + self.save_fds:
os.close(fd)
To use this you just:
with suppress_stdout_stderr():
rogue_function()
This works “pretty good”. It does suppress the printout from the rogue functions that were cluttering up my script. I noticed in testing it that it lets through raised exceptions as well as some logger print, and I’m not entirely clear why. I think it has something to do with when these messages get sent to stdout / stderr (I think it happens after my context manager exits). If anyone can confirm this, I’d be interested in hearing the details …
Answers:
If you are running this script on a linux based machine, you should be able to:
$> ./runscript.py > output.txt
Did you try to redirect stderr too?
e.g.
sys.stdout = StringIO()
sys.stderr = StringIO()
foo(bar)
sys.stdout = sys.__stdout__ # These are provided by python
sys.stderr = sys.__stderr__
Also using StringIO might use extra memory. You can use a dummy device instead (e.g. http://coreygoldberg.blogspot.com/2009/05/python-redirect-or-turn-off-stdout-and.html).
This approach (found through the related sidebar) might work. It reassigns the file descriptors rather than just the wrappers to them in sys.stdout, etc.
My solution is similar to yours but uses contextlib and is a little shorter and easier to understand (IMHO).
import contextlib
@contextlib.contextmanager
def stdchannel_redirected(stdchannel, dest_filename):
"""
A context manager to temporarily redirect stdout or stderr
e.g.:
with stdchannel_redirected(sys.stderr, os.devnull):
if compiler.has_function('clock_gettime', libraries=['rt']):
libraries.append('rt')
"""
try:
oldstdchannel = os.dup(stdchannel.fileno())
dest_file = open(dest_filename, 'w')
os.dup2(dest_file.fileno(), stdchannel.fileno())
yield
finally:
if oldstdchannel is not None:
os.dup2(oldstdchannel, stdchannel.fileno())
if dest_file is not None:
dest_file.close()
The context for why I created this is at this blog post. Similar to yours I think.
I use it like this in a setup.py:
with stdchannel_redirected(sys.stderr, os.devnull):
if compiler.has_function('clock_gettime', libraries=['rt']):
libraries.append('rt')
Not really requested by the OP, but I needed to hide and store the output, and did like follows:
from io import StringIO
import sys
class Hider:
def __init__(self, channels=('stdout',)):
self._stomach = StringIO()
self._orig = {ch : None for ch in channels}
def __enter__(self):
for ch in self._orig:
self._orig[ch] = getattr(sys, ch)
setattr(sys, ch, self)
return self
def write(self, string):
self._stomach.write(string)
def flush(self):
pass
def autopsy(self):
return self._stomach.getvalue()
def __exit__(self, *args):
for ch in self._orig:
setattr(sys, ch, self._orig[ch])
Usage:
with Hider() as h:
spammy_function()
result = h.autopsy()
(tested only with Python 3)
EDIT: now allows to select stderr, stdout or both, as in Hider([stdout, stderr])
python 3.6 working version, tested with million suppressions without any errors
import os
import sys
class suppress_stdout_stderr(object):
def __enter__(self):
self.outnull_file = open(os.devnull, 'w')
self.errnull_file = open(os.devnull, 'w')
self.old_stdout_fileno_undup = sys.stdout.fileno()
self.old_stderr_fileno_undup = sys.stderr.fileno()
self.old_stdout_fileno = os.dup ( sys.stdout.fileno() )
self.old_stderr_fileno = os.dup ( sys.stderr.fileno() )
self.old_stdout = sys.stdout
self.old_stderr = sys.stderr
os.dup2 ( self.outnull_file.fileno(), self.old_stdout_fileno_undup )
os.dup2 ( self.errnull_file.fileno(), self.old_stderr_fileno_undup )
sys.stdout = self.outnull_file
sys.stderr = self.errnull_file
return self
def __exit__(self, *_):
sys.stdout = self.old_stdout
sys.stderr = self.old_stderr
os.dup2 ( self.old_stdout_fileno, self.old_stdout_fileno_undup )
os.dup2 ( self.old_stderr_fileno, self.old_stderr_fileno_undup )
os.close ( self.old_stdout_fileno )
os.close ( self.old_stderr_fileno )
self.outnull_file.close()
self.errnull_file.close()
As of python 3.5 we can do this with minimal work using built-ins in contextlib, namely redirect_stdout and redirect_stderr. We only need to combine these two built-in context managers in a custom context manager of ours, which can be easily done using the nice pattern in Martijn’s answer here. Redirecting both outputs to os.devnull should be safe and portable enough.
from contextlib import contextmanager,redirect_stderr,redirect_stdout
from os import devnull
@contextmanager
def suppress_stdout_stderr():
"""A context manager that redirects stdout and stderr to devnull"""
with open(devnull, 'w') as fnull:
with redirect_stderr(fnull) as err, redirect_stdout(fnull) as out:
yield (err, out)
Note that suppressing stderr will still give you full tracebacks when something breaks, which is a good thing:
import sys
def rogue_function():
print('spam to stdout')
print('important warning', file=sys.stderr)
1 + 'a'
return 42
with suppress_stdout_stderr():
rogue_function()
When run the above only prints
Traceback (most recent call last):
File "tmp.py", line 20, in <module>
rogue_function()
File "foo.py", line 16, in rogue_function
1 + 'a'
TypeError: unsupported operand type(s) for +: 'int' and 'str'
to the terminal. Unhandled exceptions should never go unnoticed.
I use a decorator for this. It saves sys.stdout and sys.stderr references and makes these variables point to null. Then, after the function execution the original references are retrieved. It is important to note the try/except block, that allows the retrieval of the original references even when an exception is raised on the function.
def suppress_std(func):
def wrapper(*args, **kwargs):
stderr_tmp = sys.stderr
stdout_tmp = sys.stdout
null = open(os.devnull, 'w')
sys.stdout = null
sys.stderr = null
try:
result = func(*args, **kwargs)
sys.stderr = stderr_tmp
sys.stdout = stdout_tmp
return result
except:
sys.stderr = stderr_tmp
sys.stdout = stdout_tmp
raise
return wrapper
To use:
@suppress_std
def function_std_suppressed():
# code here
Just use Linux/Unix:
./myscript.py 2>/dev/null # gets rid of stderr
./myscript.py 2>/somewhere/myerror.log
I have a Python script that is using some closed-box Python functions (i.e. I can’t edit these functions) provided by my employer. When I call these functions, they are printing output to my linux terminal that I would like to suppress. I’ve tried redirecting stdout / stderr via;
orig_out = sys.stdout
sys.stdout = StringIO()
rogue_function()
sys.stdout = orig_out
but this fails to catch the output. I think the functions I’m calling via-Python (rogue_function() from above) are really wrappers for compiled C-code, which are actually doing the printing.
Does anyone know of a way I can do a “deep-capture” of any print handed to stdout / stderr by a function (and any sub-functions that function calls)?
UPDATE:
I ended up taking the method outlined in the selected answer below and writing a context manager to supress stdout and stderr:
# Define a context manager to suppress stdout and stderr.
class suppress_stdout_stderr(object):
'''
A context manager for doing a "deep suppression" of stdout and stderr in
Python, i.e. will suppress all print, even if the print originates in a
compiled C/Fortran sub-function.
This will not suppress raised exceptions, since exceptions are printed
to stderr just before a script exits, and after the context manager has
exited (at least, I think that is why it lets exceptions through).
'''
def __init__(self):
# Open a pair of null files
self.null_fds = [os.open(os.devnull,os.O_RDWR) for x in range(2)]
# Save the actual stdout (1) and stderr (2) file descriptors.
self.save_fds = [os.dup(1), os.dup(2)]
def __enter__(self):
# Assign the null pointers to stdout and stderr.
os.dup2(self.null_fds[0],1)
os.dup2(self.null_fds[1],2)
def __exit__(self, *_):
# Re-assign the real stdout/stderr back to (1) and (2)
os.dup2(self.save_fds[0],1)
os.dup2(self.save_fds[1],2)
# Close all file descriptors
for fd in self.null_fds + self.save_fds:
os.close(fd)
To use this you just:
with suppress_stdout_stderr():
rogue_function()
This works “pretty good”. It does suppress the printout from the rogue functions that were cluttering up my script. I noticed in testing it that it lets through raised exceptions as well as some logger print, and I’m not entirely clear why. I think it has something to do with when these messages get sent to stdout / stderr (I think it happens after my context manager exits). If anyone can confirm this, I’d be interested in hearing the details …
If you are running this script on a linux based machine, you should be able to:
$> ./runscript.py > output.txt
Did you try to redirect stderr too?
e.g.
sys.stdout = StringIO()
sys.stderr = StringIO()
foo(bar)
sys.stdout = sys.__stdout__ # These are provided by python
sys.stderr = sys.__stderr__
Also using StringIO might use extra memory. You can use a dummy device instead (e.g. http://coreygoldberg.blogspot.com/2009/05/python-redirect-or-turn-off-stdout-and.html).
This approach (found through the related sidebar) might work. It reassigns the file descriptors rather than just the wrappers to them in sys.stdout, etc.
My solution is similar to yours but uses contextlib and is a little shorter and easier to understand (IMHO).
import contextlib
@contextlib.contextmanager
def stdchannel_redirected(stdchannel, dest_filename):
"""
A context manager to temporarily redirect stdout or stderr
e.g.:
with stdchannel_redirected(sys.stderr, os.devnull):
if compiler.has_function('clock_gettime', libraries=['rt']):
libraries.append('rt')
"""
try:
oldstdchannel = os.dup(stdchannel.fileno())
dest_file = open(dest_filename, 'w')
os.dup2(dest_file.fileno(), stdchannel.fileno())
yield
finally:
if oldstdchannel is not None:
os.dup2(oldstdchannel, stdchannel.fileno())
if dest_file is not None:
dest_file.close()
The context for why I created this is at this blog post. Similar to yours I think.
I use it like this in a setup.py:
with stdchannel_redirected(sys.stderr, os.devnull):
if compiler.has_function('clock_gettime', libraries=['rt']):
libraries.append('rt')
Not really requested by the OP, but I needed to hide and store the output, and did like follows:
from io import StringIO
import sys
class Hider:
def __init__(self, channels=('stdout',)):
self._stomach = StringIO()
self._orig = {ch : None for ch in channels}
def __enter__(self):
for ch in self._orig:
self._orig[ch] = getattr(sys, ch)
setattr(sys, ch, self)
return self
def write(self, string):
self._stomach.write(string)
def flush(self):
pass
def autopsy(self):
return self._stomach.getvalue()
def __exit__(self, *args):
for ch in self._orig:
setattr(sys, ch, self._orig[ch])
Usage:
with Hider() as h:
spammy_function()
result = h.autopsy()
(tested only with Python 3)
EDIT: now allows to select stderr, stdout or both, as in Hider([stdout, stderr])
python 3.6 working version, tested with million suppressions without any errors
import os
import sys
class suppress_stdout_stderr(object):
def __enter__(self):
self.outnull_file = open(os.devnull, 'w')
self.errnull_file = open(os.devnull, 'w')
self.old_stdout_fileno_undup = sys.stdout.fileno()
self.old_stderr_fileno_undup = sys.stderr.fileno()
self.old_stdout_fileno = os.dup ( sys.stdout.fileno() )
self.old_stderr_fileno = os.dup ( sys.stderr.fileno() )
self.old_stdout = sys.stdout
self.old_stderr = sys.stderr
os.dup2 ( self.outnull_file.fileno(), self.old_stdout_fileno_undup )
os.dup2 ( self.errnull_file.fileno(), self.old_stderr_fileno_undup )
sys.stdout = self.outnull_file
sys.stderr = self.errnull_file
return self
def __exit__(self, *_):
sys.stdout = self.old_stdout
sys.stderr = self.old_stderr
os.dup2 ( self.old_stdout_fileno, self.old_stdout_fileno_undup )
os.dup2 ( self.old_stderr_fileno, self.old_stderr_fileno_undup )
os.close ( self.old_stdout_fileno )
os.close ( self.old_stderr_fileno )
self.outnull_file.close()
self.errnull_file.close()
As of python 3.5 we can do this with minimal work using built-ins in contextlib, namely redirect_stdout and redirect_stderr. We only need to combine these two built-in context managers in a custom context manager of ours, which can be easily done using the nice pattern in Martijn’s answer here. Redirecting both outputs to os.devnull should be safe and portable enough.
from contextlib import contextmanager,redirect_stderr,redirect_stdout
from os import devnull
@contextmanager
def suppress_stdout_stderr():
"""A context manager that redirects stdout and stderr to devnull"""
with open(devnull, 'w') as fnull:
with redirect_stderr(fnull) as err, redirect_stdout(fnull) as out:
yield (err, out)
Note that suppressing stderr will still give you full tracebacks when something breaks, which is a good thing:
import sys
def rogue_function():
print('spam to stdout')
print('important warning', file=sys.stderr)
1 + 'a'
return 42
with suppress_stdout_stderr():
rogue_function()
When run the above only prints
Traceback (most recent call last):
File "tmp.py", line 20, in <module>
rogue_function()
File "foo.py", line 16, in rogue_function
1 + 'a'
TypeError: unsupported operand type(s) for +: 'int' and 'str'
to the terminal. Unhandled exceptions should never go unnoticed.
I use a decorator for this. It saves sys.stdout and sys.stderr references and makes these variables point to null. Then, after the function execution the original references are retrieved. It is important to note the try/except block, that allows the retrieval of the original references even when an exception is raised on the function.
def suppress_std(func):
def wrapper(*args, **kwargs):
stderr_tmp = sys.stderr
stdout_tmp = sys.stdout
null = open(os.devnull, 'w')
sys.stdout = null
sys.stderr = null
try:
result = func(*args, **kwargs)
sys.stderr = stderr_tmp
sys.stdout = stdout_tmp
return result
except:
sys.stderr = stderr_tmp
sys.stdout = stdout_tmp
raise
return wrapper
To use:
@suppress_std
def function_std_suppressed():
# code here
Just use Linux/Unix:
./myscript.py 2>/dev/null # gets rid of stderr
./myscript.py 2>/somewhere/myerror.log