Python default and multi level logging

Question

From what I read and understood, Python logging module by default logs to stderr.

If I run this python code:

import logging


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

as

python main.py 1> stdout.txt 2> stderr.txt

I get my logs in stderr.txt and nothing in stdout.txt – my logs are redirected to stderr.

This default behaviour is problematic when logs are streamed to logging aggregation services such as datadog or papertrail. Since its streamed to stderr, the logs are marked as errors when in reality they are not.

So I tried to create multiple log handlers as follows:

import logging
import sys


stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stderrHandler = logging.StreamHandler(stream=sys.stderr)

logging.basicConfig(level=logging.DEBUG, handlers=[stdoutHandler, stderrHandler])

stdoutHandler.setLevel(logging.DEBUG)
stderrHandler.setLevel(logging.ERROR)


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

When I run this code, I get errors in sterr.txt but also all the logs in stdout.txt – I ended up having log duplication error logs appear in both the stderr and stdout streams.

Is there a better way to handle the differentiation of error logs from the rest in Python?

I tried loguru package as well, also no luck in stream separation…
Thanks in advance

Asked By: Pavel Durov

||

Source

Answer 1

You can probably benefit from the approach described here in the official documentation.

Answered By: Vinay Sajip

Answer 2

Following the approach described in the official documentation:
https://docs.python.org/3.10/howto/logging-cookbook.html#custom-handling-of-levels

Here’s my code with stream segregation:

import sys
import logging



ROOT_LEVEL = logging.DEBUG


class StdoutFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= ROOT_LEVEL and record.levelno < logging.ERROR


class StderrFilter(logging.Filter):
    def filter(self, record):
        return record.levelno >= logging.ERROR



stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stdoutHandler.addFilter(StdoutFilter())

stderrHandler = logging.StreamHandler(stream=sys.stderr)
stderrHandler.addFilter(StderrFilter())


root_logger = logging.getLogger()
root_logger.setLevel(ROOT_LEVEL)

root_logger.addHandler(stderrHandler)
root_logger.addHandler(stdoutHandler)

Answered By: Pavel Durov

Python default and multi level logging

Question:

Answers: