Python, want logging with log rotation and compression


Can anyone suggest a way in python to do logging with:

  • log rotation every day
  • compression of logs when they’re rotated
  • optional – delete oldest log file to preserve X MB of free space
  • optional – sftp log files to server

Thanks for any responses,

Asked By: fred basset



  • log rotation every day: Use a TimedRotatingFileHandler
  • compression of logs: Set the encoding='bz2' parameter. (Note this “trick” will only work for Python2. ‘bz2’ is no longer considered an encoding in Python3.)
  • optional – delete oldest log file to preserve X MB of free space.
    You could (indirectly) arrange this using a RotatingFileHandler. By setting the maxBytes parameter, the log file will rollover when it reaches a certain size. By setting the backupCount parameter, you can control how many rollovers are kept. The two parameters together allow you to control the maximum space consumed by the log files. You could probably subclass the TimeRotatingFileHandler to incorporate this behavior into it as well.

Just for fun, here is how you could subclass TimeRotatingFileHandler. When you run the script below, it will write log files to /tmp/log_rotate*.

With a small value for time.sleep (such as 0.1), the log files fill up quickly, reach the maxBytes limit, and are then rolled over.

With a large time.sleep (such as 1.0), the log files fill up slowly, the maxBytes limit is not reached, but they roll over anyway when the timed interval (of 10 seconds) is reached.

All the code below comes from logging/ I simply meshed TimeRotatingFileHandler with RotatingFileHandler in the most straight-forward way possible.

import time
import re
import os
import stat
import logging
import logging.handlers as handlers

class SizedTimedRotatingFileHandler(handlers.TimedRotatingFileHandler):
    Handler for logging to a set of files, which switches from one file
    to the next when the current file reaches a certain size, or at certain
    timed intervals

    def __init__(self, filename, maxBytes=0, backupCount=0, encoding=None,
                 delay=0, when='h', interval=1, utc=False):
            self, filename, when, interval, backupCount, encoding, delay, utc)
        self.maxBytes = maxBytes

    def shouldRollover(self, record):
        Determine if rollover should occur.

        Basically, see if the supplied record would cause the file to exceed
        the size limit we have.
        if is None:                 # delay was set...
   = self._open()
        if self.maxBytes > 0:                   # are we rolling over?
            msg = "%sn" % self.format(record)
            # due to non-posix-compliant Windows feature
  , 2)
            if + len(msg) >= self.maxBytes:
                return 1
        t = int(time.time())
        if t >= self.rolloverAt:
            return 1
        return 0

def demo_SizedTimedRotatingFileHandler():
    log_filename = '/tmp/log_rotate'
    logger = logging.getLogger('MyLogger')
    handler = SizedTimedRotatingFileHandler(
        log_filename, maxBytes=100, backupCount=5,
        when='s', interval=10,
        # encoding='bz2',  # uncomment for bz2 compression
    for i in range(10000):
        logger.debug('i=%d' % i)

Answered By: unutbu

In addition to unutbu’s answer: here’s how to modify the TimedRotatingFileHandler to compress using zip files.

import logging
import logging.handlers
import zipfile
import codecs
import sys
import os
import time
import glob

class TimedCompressedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
    Extended version of TimedRotatingFileHandler that compress logs on rollover.
    def doRollover(self):
        do a rollover; in this case, a date/time stamp is appended to the filename
        when the rollover happens.  However, you want the file to be named for the
        start of the interval, not the current time.  If there is a backup count,
        then we have to get a list of matching filenames, sort them and remove
        the one with the oldest suffix.
        # get the time that this sequence started at and make it a TimeTuple
        t = self.rolloverAt - self.interval
        timeTuple = time.localtime(t)
        dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
        if os.path.exists(dfn):
        os.rename(self.baseFilename, dfn)
        if self.backupCount > 0:
            # find the oldest log file and delete it
            s = glob.glob(self.baseFilename + ".20*")
            if len(s) > self.backupCount:
        #print "%s -> %s" % (self.baseFilename, dfn)
        if self.encoding:
   =, 'w', self.encoding)
   = open(self.baseFilename, 'w')
        self.rolloverAt = self.rolloverAt + self.interval
        if os.path.exists(dfn + ".zip"):
            os.remove(dfn + ".zip")
        file = zipfile.ZipFile(dfn + ".zip", "w")
        file.write(dfn, os.path.basename(dfn), zipfile.ZIP_DEFLATED)

if __name__=='__main__':
    ## Demo of using TimedCompressedRotatingFileHandler() to log every 5 seconds,
    ##     to one uncompressed file and five rotated and compressed files

    os.nice(19)   # I always nice test code

    logHandler = TimedCompressedRotatingFileHandler("mylog", when="S",
        interval=5, backupCount=5) # Total of six rotated log files, rotating every 5 secs
    logFormatter = logging.Formatter(
        fmt='%(asctime)s.%(msecs)03d %(message)s', 
        datefmt='%Y-%m-%d %H:%M:%S'
    mylogger = logging.getLogger('MyLogRef')

    # Write lines non-stop into the logger and rotate every 5 seconds
    ii = 0
    while True:
        mylogger.debug("Test {0}".format(ii))
        ii += 1
Answered By: Martijn Rutten

The other way to compress logfile during rotate (new in python 3.3) is using BaseRotatingHandler (and all inherited) class attribute rotator for example:

import gzip
import os
import logging
import logging.handlers

class GZipRotator:
    def __call__(self, source, dest):
        os.rename(source, dest)
        f_in = open(dest, 'rb')
        f_out ="%s.gz" % dest, 'wb')

logformatter = logging.Formatter('%(asctime)s;%(levelname)s;%(message)s')
log = logging.handlers.TimedRotatingFileHandler('debug.log', 'midnight', 1, backupCount=5)
log.rotator = GZipRotator()

logger = logging.getLogger('main')


More you can see here.

Answered By: evgenek

To copy the file, gzip the copied file (using epoch time), and then clearing out the existing file in a way that won’t upset the logging module:

import gzip
import logging
import os
from shutil import copy2
from time import time

def logRoll(logfile_name):
    log_backup_name = logfile_name + '.' + str(int(time()))
        copy2(logfile_name, log_backup_name)   
    except IOError, err:
        logging.debug(' No logfile to roll')
    f_in = open(log_backup_name, 'rb')
    f_out = + '.gz', 'wb')
    f=open(logfile_name, 'w')
Answered By: David

I guess it’s too late to join the party, but here is what I did. I created a new class inheriting logging.handlers.RotatingFileHandler class and added a couple of lines to gzip the file before moving it.

#!/usr/bin/env python

# Import all the needed modules
import logging.handlers
import sys
import time
import gzip
import os
import shutil
import random
import string

__version__ = 1.0
__descr__ = "This logic is written keeping in mind UNIX/LINUX/OSX platforms only"

# Create a new class that inherits from RotatingFileHandler. This is where we add the new feature to compress the logs
class CompressedRotatingFileHandler(logging.handlers.RotatingFileHandler):
    def doRollover(self):
        Do a rollover, as described in __init__().
        if self.backupCount > 0:
            for i in range(self.backupCount - 1, 0, -1):
                sfn = "%s.%d.gz" % (self.baseFilename, i)
                dfn = "%s.%d.gz" % (self.baseFilename, i + 1)
                if os.path.exists(sfn):
                    # print "%s -> %s" % (sfn, dfn)
                    if os.path.exists(dfn):
                    os.rename(sfn, dfn)
            dfn = self.baseFilename + ".1.gz"
            if os.path.exists(dfn):
            # These two lines below are the only new lines. I commented out the os.rename(self.baseFilename, dfn) and
            #  replaced it with these two lines.
            with open(self.baseFilename, 'rb') as f_in,, 'wb') as f_out:
                shutil.copyfileobj(f_in, f_out)
            # os.rename(self.baseFilename, dfn)
            # print "%s -> %s" % (self.baseFilename, dfn)
        self.mode = 'w' = self._open()

# Specify which file will be used for our logs
log_filename = "/Users/myname/Downloads/test_logs/sample_log.txt"

# Create a logger instance and set the facility level
my_logger = logging.getLogger()

# Create a handler using our new class that rotates and compresses
file_handler = CompressedRotatingFileHandler(filename=log_filename, maxBytes=1000000, backupCount=10)

# Create a stream handler that shows the same log on the terminal (just for debug purposes)
view_handler = logging.StreamHandler(stream=sys.stdout)

# Add all the handlers to the logging instance

# This is optional to beef up the logs
random_huge_data = "".join(random.choice(string.ascii_letters) for _ in xrange(10000))

# All this code is user-specific, write your own code if you want to play around
count = 0
while True:
    my_logger.debug("This is the message number %s" % str(count))
    count += 1
    if count % 100 == 0:
        count = 0
Answered By: gixxer

I think that the best option will be to use current implementation of TimedRotatingFileHandler and after renaming log file to the rotated version just compress it:

import zipfile
import os
from logging.handlers import TimedRotatingFileHandler

class TimedCompressedRotatingFileHandler(TimedRotatingFileHandler):
    Extended version of TimedRotatingFileHandler that compress logs on rollover.
    def find_last_rotated_file(self):
        dir_name, base_name = os.path.split(self.baseFilename)
        file_names = os.listdir(dir_name)
        result = []
        prefix = '{}.20'.format(base_name)  # we want to find a rotated file with eg filename.2017-12-12... name
        for file_name in file_names:
            if file_name.startswith(prefix) and not file_name.endswith('.zip'):
        return result[0]

    def doRollover(self):
        super(TimedCompressedRotatingFileHandler, self).doRollover()

        dfn = self.find_last_rotated_file()
        dfn_zipped = '{}.zip'.format(dfn)
        if os.path.exists(dfn_zipped):
        with zipfile.ZipFile(dfn_zipped, 'w') as f:
            f.write(dfn, dfn_zipped, zipfile.ZIP_DEFLATED)
Answered By: kamilglod

Be warned: The class signatures have changed in python 3. Here is my working example for python 3.6

import logging.handlers
import os
import zlib

def namer(name):
    return name + ".gz"

def rotator(source, dest):
    print(f'compressing {source} -> {dest}')
    with open(source, "rb") as sf:
        data =
        compressed = zlib.compress(data, 9)
        with open(dest, "wb") as df:

err_handler = logging.handlers.TimedRotatingFileHandler('/data/errors.log', when="M", interval=1,
                                                        encoding='utf-8', backupCount=30, utc=True)
err_handler.rotator = rotator
err_handler.namer = namer

logger = logging.getLogger("Rotating Log")

Answered By: David Dehghan

Here is my solution(modified from evgenek), simple and does not block python code while gzipping huge log files:

class GZipRotator:
    def __call__(self, source, dest):
        os.rename(source, dest)
        subprocess.Popen(['gzip', dest])
Answered By: Yuriy

I have added below one solution where i am basically zipping old
backup logs to zip with timestamp on it. using one extra variable
called ZipbackupCount. # no of old files to be zipped

e.g. we have logs like this. (backupcount = 5 and ZipbackupCount = 2)





once count of backup logs hits to 5 it triggers to zip a.log.5 and a.log.4 to above zip and continues.

import os
import datetime
import gzip
import logging.handlers
import zipfile


class NewRotatingFileHandler(logging.handlers.RotatingFileHandler):
    def __init__(self, filename, **kws):
        backupCount = kws.get('backupCount', 0)
        self.backup_count = backupCount
        self.ZipbackupCount = kws.pop('ZipbackupCount', 0)
        self.file_name = filename
        self.log_dir = os.path.split(self.file_name)[0]
        self.log_file_name = os.path.split(self.file_name)[-1]

        logging.handlers.RotatingFileHandler.__init__(self, filename, **kws)

    def doArchive(self, old_log):
        with open(old_log) as log:
            with + '.gz', 'wb') as comp_log:


    def doRollover(self):
        super(NewRotatingFileHandler, self).doRollover()

        zip_file_name = self.log_file_name + "." +"%d-%m-%Y-%H-%M-%S") + ".zip"

        if os.path.exists(self.rotation_filename("%s.%d" % (self.baseFilename, self.backupCount))) and self.ZipbackupCount > 0 and self.file_name:

            with zipfile.ZipFile(os.path.join(self.log_dir, zip_file_name), "w", zipfile.ZIP_DEFLATED, allowZip64=True) as zf:

                for i in range(self.backupCount, self.backupCount - self.ZipbackupCount, -1):

                    sfn = self.rotation_filename("%s.%d" % (self.baseFilename, i))

                    if os.path.exists(sfn):
                        zf.write(sfn, "%s.%d" % (self.log_file_name, i))


# handler = NewRotatingFileHandler(filename=os.path.join(PROJECT_PATH, LOG_DIR, LOG_FILE_NAME),
#                                  maxBytes=LOG_FILESIZE, backupCount=5, ZipbackupCount=2)
# handler.doRollover()

Answered By: Lalit Suthar

I see lot of answers focussing on overriding the doRollover method of the handler class, which is fine and works but I think a much cleaner and recommended solution is to define the rotator attribute on the handler class which is checked for in the rotate method rather than overriding doRollover.

def setup_logger():
    def bz2namer(name):
        return name + ".bz2"

    def bzip_rotator(source, dest):
        with open(source, "rb") as sf:
            data =
            compressed = bz2.compress(data, 9)
            with open(dest, "wb") as df:

    log = logging.getLogger("myapp")
    file_handler = RotatingFileHandler(filename="app.log", backupCount=10, maxBytes=10000000)
    file_handler.rotator = bzip_rotator
    file_handler.namer = bz2namer

    _log_format = "%(asctime)s [%(process)d] [%(levelname)s] [%(filename)s: %(funcName)s] - %(message)s "
    log_formatter = logging.Formatter(_log_format, datefmt="%Y-%m-%d %H:%M:%S")
    log.propagate = False
Answered By: Rohit

Since python 3.3 you can easily extend any subclass of BaseRotatingFileHandler to implement gzipping:

import os
from logging.handlers import TimedRotatingFileHandler
from gzip import open as gzip_open

class CompressingTimedRotatingFileHandler(TimedRotatingFileHandler):
    """TimedRotatingFileHandler with gzip compression on rotate"""

    def rotation_filename(self, default_name: str) -> str:
        Extend the default filename to use gzip-ending.
        :param default_name: The default filename of this handler
        return default_name + ".gz"

    def rotate(self, source: str, dest: str) -> None:
        Rotate the current log
        :param source: The source filename. This is normally the base
                       filename, e.g. 'test.log'
        :param dest:   The destination filename. This is normally
                       what the source is rotated to, e.g. 'test.log.1'.
        # compress source file and write to destination
        with open(source, 'rb') as f_in, gzip_open(dest, 'wb') as f_out:
        # delete source file
Answered By: der-joel
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.