python logger changing it's own log level after error

Question

I have tried to write a script here that supports different log levels for the stream and file handlers. Initially I’ve set the log level for the stream hander to be ERROR and file to be INFO, however, after the first error, the stream handler is reporting at the DEBUG level and not with the format I’ve specified. It would seem it’s created a new stream handler somehow? as future logging at the previously specified level for the stream hander are now duplicated. Here’s the relevant portion of my code:

args = parser.parse_args()
s = args.source
d = args.destination
group = args.group
debug = args.debug
interactive = args.interactive

def set_log_level_from_verbose(args, l):
    if not args.verbose:
        l.setLevel(logging.ERROR)
    elif args.verbose == 1:
        l.setLevel(logging.WARNING)
    elif args.verbose == 2:
        l.setLevel(logging.INFO)
    elif args.verbose >= 2:
        l.setLevel(logging.DEBUG)
    else:
        l.critical("UNEXPLAINED NEGATIVE VERBOSITY COUNT!")

logger = logging.getLogger('DOIT_DATA_PROMOTION')
logger.setLevel(logging.DEBUG)

# create a console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
set_log_level_from_verbose(args, ch)

# add log message handler to the logger
fh = logging.handlers.TimedRotatingFileHandler('DOIT_DATA_PROMOTION.log', when='midnight', backupCount=5)
fh.setLevel(logging.INFO)

# create formatter and add it to the handlers
fmt = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
fmt1 = logging.Formatter('%(levelname)s - %(message)s')
fh.setFormatter(fmt)
ch.setFormatter(fmt1)

# Add handers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.info("n++++++++++++++++ Start Operations ++++++++++++++++")

logger.error("Sample ERROR")
logger.info("New INFO")

running this with args.verbose not set and console output in this case will show

ERROR - Sample ERROR
INFO:DOIT_DATA_PROMOTION:NewInfo

I’m sure it’s a dumb error but does anyone else see why this is happening?

Asked By: ShaunLangley

||

Source

Answer 1

It looks as if something might be calling logging.basicConfig() which would set up a StreamHandler for the root logger, and explain what you’re seeing.

The basicConfig() call is either explicit somewhere in the code you call (not necessarily what’s shown in your snippet above) or via accidentally calling logging.debug(...) [or similar] instead of logger.debug(...) [or similar].

Answered By: Vinay Sajip

Answer 2

I am having the same issue but I don’t think logging.basicConfig() can cause this, unless its force option was used. I found out that in my case it was the tqdm module that was resetting the stream handler level (as can be seen in this PR). Here’s my logging setup, hopefully this helps anyone with the same issue…

def setup_logging(logfile: Path=Path(), debug: bool=False):
    """
    Setup the logging framework
    :param logfile:     Name of the logfile
    :param debug:       Add the function name to the log-messages and set the console logging level to VERBOSE
    :return:
     """

    # Set the default formats
    if debug:
        fmt  = '%(asctime)s - %(name)s - %(levelname)s | %(message)s'
        cfmt = '%(levelname)s - %(name)s | %(message)s'
    else:
        fmt  = '%(asctime)s - %(levelname)s | %(message)s'
        cfmt = '%(levelname)s | %(message)s'
    datefmt  = '%Y-%m-%d %H:%M:%S'

    # Add a verbose logging level = 15
    logging.VERBOSE = 15
    logging.addLevelName(logging.VERBOSE, 'VERBOSE')
    logging.__all__ += ['VERBOSE'] if 'VERBOSE' not in logging.__all__ else []
    def verbose(self, message, *args, **kws):
        if self.isEnabledFor(logging.VERBOSE): self._log(logging.VERBOSE, message, args, **kws)
    logging.Logger.verbose = verbose

    # Add a succes logging level = 25
    logging.SUCCESS = 25
    logging.addLevelName(logging.SUCCESS, 'SUCCESS')
    logging.__all__ += ['SUCCESS'] if 'SUCCESS' not in logging.__all__ else []
    def success(self, message, *args, **kws):
        if self.isEnabledFor(logging.SUCCESS): self._log(logging.SUCCESS, message, args, **kws)
    logging.Logger.success = success

    # Set the root logging level
    logger = logging.getLogger()
    logger.setLevel('VERBOSE')

    # Add the console streamhandler and bring some color to those boring logs! :-)
    coloredlogs.install(level='VERBOSE' if debug or not logfile.name else 'INFO', fmt=cfmt, datefmt=datefmt)   # NB: Using tqdm sets the streamhandler level to 0, see: https://github.com/tqdm/tqdm/pull/1235
    coloredlogs.DEFAULT_LEVEL_STYLES['verbose']['color'] = 245  # = Gray

    if not logfile.name:
        return

    # Add the log filehandler
    logfile.parent.mkdir(parents=True, exist_ok=True)      # Create the log dir if it does not exist
    formatter  = logging.Formatter(fmt=fmt, datefmt=datefmt)
    loghandler = logging.FileHandler(logfile)
    loghandler.setLevel('VERBOSE')
    loghandler.setFormatter(formatter)
    loghandler.set_name('loghandler')
    logger.addHandler(loghandler)

    # Add the error/warnings filehandler
    errorhandler = logging.FileHandler(logfile.with_suffix('.errors'), mode='w')
    errorhandler.setLevel('WARNING')
    errorhandler.setFormatter(formatter)
    errorhandler.set_name('errorhandler')
    logger.addHandler(errorhandler)

Answered By: Marcel Zwiers

python logger changing it's own log level after error

Question:

Answers: