How can I capture the console output of installed python packages and send them to my logger?
Question:
I’m developing a python application and I want to have a nice console output when running the package. This is ruined by warnings, print statements, or other forms of console output inherited from installed packages (like matplotlib
).
Example:
> python test.py
[10/04/2022 01:38:15 AM] INFO Creating directory simulation.py:891
INFO Saving GIF... simulation.py:793
INFO Finished saving GIF simulation.py:803
INFO Saving graph... simulation.py:816
/blah/blah/blah/python3.9/site-packages/matplotlib/cbook/__init__.py:1026: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x = np.asanyarray(x)
[10/04/2022 01:38:16 AM] INFO Finished saving graph simulation.py:827
So all of my log messages are captured by my configured logger across all of my modules, but then the console output is interrupted by ugly messages from installed packages that I don’t care about and can’t change.
How do I send these messages (or all console output?) to my logger so that they are in the same format as my log messages?
This happens across many different packages so I am not looking for a solution just for matplotlib
.
Answers:
You could redirect sys.stdout
and sys.stderr
to a logger, as described in the documentation here.
Update: You can only redirect Python-level things using sys.stderr
– the example in the comment is a C-level write. To redirect that, you’ll need to use ctypes
and the approach outlined in this post in the section entitled "Redirecting with file descriptor duplication". Though I would log an issue with that library’s maintainers, as outputting to stderr from a library isn’t terribly good practice.
I’m developing a python application and I want to have a nice console output when running the package. This is ruined by warnings, print statements, or other forms of console output inherited from installed packages (like matplotlib
).
Example:
> python test.py
[10/04/2022 01:38:15 AM] INFO Creating directory simulation.py:891
INFO Saving GIF... simulation.py:793
INFO Finished saving GIF simulation.py:803
INFO Saving graph... simulation.py:816
/blah/blah/blah/python3.9/site-packages/matplotlib/cbook/__init__.py:1026: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
x = np.asanyarray(x)
[10/04/2022 01:38:16 AM] INFO Finished saving graph simulation.py:827
So all of my log messages are captured by my configured logger across all of my modules, but then the console output is interrupted by ugly messages from installed packages that I don’t care about and can’t change.
How do I send these messages (or all console output?) to my logger so that they are in the same format as my log messages?
This happens across many different packages so I am not looking for a solution just for matplotlib
.
You could redirect sys.stdout
and sys.stderr
to a logger, as described in the documentation here.
Update: You can only redirect Python-level things using sys.stderr
– the example in the comment is a C-level write. To redirect that, you’ll need to use ctypes
and the approach outlined in this post in the section entitled "Redirecting with file descriptor duplication". Though I would log an issue with that library’s maintainers, as outputting to stderr from a library isn’t terribly good practice.