How can I capture the console output of installed python packages and send them to my logger?

Question:

I’m developing a python application and I want to have a nice console output when running the package. This is ruined by warnings, print statements, or other forms of console output inherited from installed packages (like matplotlib).

Example:

> python test.py

[10/04/2022 01:38:15 AM] INFO     Creating directory                             simulation.py:891                                                                
                         INFO     Saving GIF...                                  simulation.py:793
                         INFO     Finished saving GIF                            simulation.py:803
                         INFO     Saving graph...                                simulation.py:816
/blah/blah/blah/python3.9/site-packages/matplotlib/cbook/__init__.py:1026: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  x = np.asanyarray(x)
[10/04/2022 01:38:16 AM] INFO     Finished saving graph                          simulation.py:827

So all of my log messages are captured by my configured logger across all of my modules, but then the console output is interrupted by ugly messages from installed packages that I don’t care about and can’t change.

How do I send these messages (or all console output?) to my logger so that they are in the same format as my log messages?

This happens across many different packages so I am not looking for a solution just for matplotlib.

Asked By: Micky-D

||

Answers:

You could redirect sys.stdout and sys.stderr to a logger, as described in the documentation here.

Update: You can only redirect Python-level things using sys.stderr – the example in the comment is a C-level write. To redirect that, you’ll need to use ctypes and the approach outlined in this post in the section entitled "Redirecting with file descriptor duplication". Though I would log an issue with that library’s maintainers, as outputting to stderr from a library isn’t terribly good practice.

Answered By: Vinay Sajip
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.