hourly log rotation in python / django

Question

as I checked the Python doc at python logging and with some experiment, I don’t quite under stand,

When computing the next rollover time for the first time (when the handler is created), the last modification time of an existing log file, or else the current time, is used to compute when the next rotation will occur.

I found the rotation time for a hourly rotation is affected by the time I start logging, say 12:23:33 starts, and next rotation at 13:23:33, which will finally be confusing to the log file name.

Code would be like ,

TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False)

Any way to force the hourly log rotation starts from 00 minute like 13:00:00 rather than the time the logging starts, and every log will contain only the logs within the hour its log file name indicates?

Asked By: Jason Xu

||

Source

Answer 1

Looking at the code for TimedRotatingFileHandler, you can’t force it to rotate at a specific minute value: as per the documentation, the next rollover will occur at either logfile last modification time + interval if the logfile already exists, or current time + interval if it doesn’t.

But since you seem to know the filename, you could trick TimedRotatingFileHandler by first setting the last modification time of the logfile to the current hour:

from    datetime import datetime
import  os, time

thishour = datetime.now().replace(minute = 0, second = 0, microsecond = 0)
timestamp = time.mktime(thishour.timetuple())

# this opens/creates the logfile...
with file(filename, 'a'):
    # ...and sets atime/mtime
    os.utime(filename, (timestamp, timestamp))

TimedRotatingFileHandler(filename, ...)

(untested)

Answered By: robertklep

Answer 2

I also attach another answer. Python logging support WatchedFileHandler, which will close and re-open log file with same file name if it find file info is updated. It works nicely with system level log rotate like linux logrotate daemon.

Answered By: Jason Xu

Answer 3

    logger = logging.getLogger()

    log_format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s'
    logging.basicConfig(format=log_format, level=level)

    # Create 'log' directory if not present
    log_path = os.path.dirname(logfile)
    if not os.path.exists(log_path):
        os.makedirs(log_path)

    handler = TimedRotatingFileHandler(
        logfile,
        when="H",
        interval=1,
        encoding="utf-8")

    handler.setFormatter(logging.Formatter(log_format))
    handler.extMatch = re.compile(r"^d{8}$")
    handler.suffix = "%Y%m%d"
    handler.setLevel(level)

    logger.addHandler(handler)

Answered By: Endogen

Answer 4

I wrote a custom Handler, through which you can achieve the functionality you need.

import logging
import os
from custom_rotating_file_handler import CustomRotatingFileHandler

def custom_rotating_logger(log_fpath, when='H'):
    os.makedirs(os.path.dirname(log_fpath), exist_ok=True)
    file_handler = CustomRotatingFileHandler(log_fpath,
                                             when=when,
                                             backupCount=10,
                                             encoding="utf-8")

    formatter = logging.Formatter('%(asctime)s %(message)s')
    file_handler.setFormatter(formatter)

    logger = logging.getLogger('MyCustomLogger')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(file_handler)
    return logger


# For test: log rotating at beginning of each minute.
logger = custom_rotating_logger('/tmp/tmp.log', when='M')

import time
for i in range(100):
    time.sleep(1)
    logger.debug('ddddd')
    logger.info('iiiii')
    logger.warning('wwwww')

Save the code above as testhandler.py and run the following command:

python3 testhandler.py && sudo find /tmp -name 'tmp.log.*'

You can see the output like this:

/tmp/tmp.log.202212142132
/tmp/tmp.log.202212142133

Content in the log file:

> head -n 3 /tmp/tmp.log.202212142133
2022-12-14 21:33:00,483 ddddd
2022-12-14 21:33:00,490 iiiii
2022-12-14 21:33:00,490 wwwww

Answered By: Wong

hourly log rotation in python / django

Question:

Answers: