Is this python logging module bug while using TimedRotatingFileHandler and rollover by hour?

Question

Using python logging module class TimedRotatingFileHandler. The class check should rollover use below,

def shouldRollover(self, record):
    t = int(time.time())
    if t >= self.rolloverAt:
        return 1
    return 0

and it compute next rollover time by using

def doRollover(self): 
    currentTime = int(time.time())
    newRolloverAt = self.computeRollover(currentTime)
    self.rolloverAt = newRolloverAt
def computeRollover(self, currentTime):
    result = currentTime + self.interval

For example, if process started at 2016/03/01 19:33:00, then the self.rolloverAt first set to 2016/03/01 20:00:00, but may no log to write until 2016/03/01 20:00:05, so the first rollover will happen at 20:00:05, and set the newRolloverAt to 2016/03/01 21:00:05, then the next all rollover will not happen at the begin of one hour.

Asked By: ccwenlin

||

Source

Answer 1

No, this is not a bug. This is by design instead. If you adhere to exact intervals from the starting time, you can end up with erratic roll-overs. If your app is quiet until shortly before the next ‘fixed’ rollover time, you’d get two new files closely together (one created at 5 seconds before the hour, the next at the hour, for example).

So for the interval types S, M, H and D, sparse logging will result in a new interval starting when the next log entry comes in. That way you still get a full duration before the next rotation.

An exception is made for midnight and the W0 .. W6 weekday rotations; just below the code you found it makes a precise calculation for the next midnight (of the correct weekday if needed) for the next rollover. That’s because these intervals do imply a specific time in the day. And with whole days for logging to take place, a last-minute log entry before a rotation point is far less of a problem.

Answered By: Martijn Pieters

Answer 2

Realy this TimedRotatingFileHandler from logging API doesn’t support the behavior expected by you (and by me).
I see the sources, and I guess it was bad design, then it’s very difficult to adjust without workaround to having the behavior.
In Java, the log4j support this behavior without problems, and until now I can’t find another Python API that does it.

But I did a workaround class works like as TimeBasedTriggeringPolicy from log4j:

from logging.handlers import TimedRotatingFileHandler

class SyncTimedRotatingFileHandler(TimedRotatingFileHandler):
    def __init__(self, filename, when='h', backupCount=0, encoding=None, delay=False, utc=False, atTime=None):
        TimedRotatingFileHandler.__init__(self, filename, when=when, interval=1, backupCount=backupCount, encoding=encoding, delay=delay, utc=utc, atTime=atTime)    

    def computeRollover(self, currentTime):
        currentTime = currentTime if (self.when == 'MIDNIGHT' or self.when.startswith('W')) else currentTime - (int(currentTime) % self.interval)
        return TimedRotatingFileHandler.computeRollover(self, currentTime)

And to test:

logger = logging.Logger('myLogger')
handler = SyncTimedRotatingFileHandler('synctime.log', 'M', 0, 'UTF-8', False, False, None)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
logger.addHandler(handler)
c = 0
while (True):
  c += 1
  logger.info("Logger Test " + str(c))
  time.sleep(1)

Answered By: Fabrício Pereira

Answer 3

I wrote a custom Handler, through which you can achieve the functionality you need.

import logging
import os
from custom_rotating_file_handler import CustomRotatingFileHandler

def custom_rotating_logger(log_fpath, when='H'):
    os.makedirs(os.path.dirname(log_fpath), exist_ok=True)
    file_handler = CustomRotatingFileHandler(log_fpath,
                                             when=when,
                                             backupCount=10,
                                             encoding="utf-8")

    formatter = logging.Formatter('%(asctime)s %(message)s')
    file_handler.setFormatter(formatter)

    logger = logging.getLogger('MyCustomLogger')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(file_handler)
    return logger


# For test: log rotating at beginning of each minute.
logger = custom_rotating_logger('/tmp/tmp.log', when='M')

import time
for i in range(100):
    time.sleep(1)
    logger.debug('ddddd')
    logger.info('iiiii')
    logger.warning('wwwww')

Save the code above as testhandler.py and run the following command:

python3 testhandler.py && sudo find /tmp -name 'tmp.log.*'

You can see the output like this:

/tmp/tmp.log.202212142132
/tmp/tmp.log.202212142133

Content in the log file:

> head -n 3 /tmp/tmp.log.202212142133
2022-12-14 21:33:00,483 ddddd
2022-12-14 21:33:00,490 iiiii
2022-12-14 21:33:00,490 wwwww

Answered By: Wong

Is this python logging module bug while using TimedRotatingFileHandler and rollover by hour?

Question:

Answers: