Python Scheduled Log Rotating
Question:
EDIT:
Looks like other people are having a similar issue with TimedRotatingFileHandler.
Why doesn't my TimedRotatingFileHandler rotate at midnight?
Apparently, the logs don’t rotate unless there is some activity happening in the logs. Is there a way I can achieve the scheduled rotating functionality I want using the builtin Handlers without having to create a custom rotator?
ORIGINAL POST:
I’m using the TimedRotatingFileHandler of Python’s logging module to rotate my logs regularly.
I’ve specified the configurations in logging.conf for my app’s loggers and handlers as such:
[logger_worker]
level=DEBUG
propogate=0
qualname=worker
handlers=workerHandler
[handler_workerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=standardFormatter
args=("/var/log/app/worker.log","M",1,30,None,False,False)
- Note: for testing purposes I’ve configured the handler to rotate the logs on every minute, but ideally the logs will be rotated on a daily basis at midnight.
In my app, I am creating the logger like this:
logging.config.fileConfig("logging.conf")
log = logging.getLogger("worker")
log.debug('Hello world')
It is not working as I expected it to work:
- It is not rotating all the logs
- It is not rotating every minute as it is configured to do
Observe the ls -l output of the log directory:
-rw-r--r-- 1 root root 0 Apr 19 18:22 dpv.log
-rw-r----- 1 root root 5092 Apr 20 11:47 emperor.log
-rw-r--r-- 1 root root 88939 Apr 20 11:47 uwsgi.log
-rw-r--r-- 1 root root 494 Apr 20 11:46 worker.log
-rw-r--r-- 1 root root 45906 Apr 20 02:08 worker.log.2016-04-20_02-08
-rw-r--r-- 1 root root 494 Apr 20 11:34 worker.log.2016-04-20_11-34
-rw-r--r-- 1 root root 494 Apr 20 11:36 worker.log.2016-04-20_11-36
-rw-r--r-- 1 root root 494 Apr 20 11:44 worker.log.2016-04-20_11-44
What am I doing wrong? Is it possible to rotate the logs on a scheduled term even when nothing is being written to the logs?
Answers:
Seems TimedRotatingFileHandler doesn’t have the functionality that you need.
You can try the following function for rotating the log files at the certain time.
def log_rollover(folder_path):
# folder_path: the absolute path to your log folder
today = str(datetime.date.today()).replace('-', '')
list_of_files = os.listdir(folder_path)
for log_file in list_of_files:
file_path = folder_path + '/' + log_file
if log_file.endswith('.log') and os.stat(file_path).st_size > 0:
new_file_path = file_path.split('.')[-2] + '-' + today + '.log'
subprocess.check_call(['cp', file_path, new_file_path])
subprocess.check_call(['gzip', new_file_path])
subprocess.check_call(['cp', '/dev/null', file_path])
Also you need to use a cron job to run the function at the certain time.
APScheduler can be a good option. For example if you want to run the cron job at 3 AM, you can use:
folderPath = '/var/log/app' # The path to your log folder
scheduler = BlockingScheduler()
scheduler.add_job(log_rollover, trigger='cron', args=[folderPath], hour=3, minute=0)
scheduler.start()
Don’t forget to set your timezone for APScheduler.
I had the same issue and came up with a different approach:
from datetime import datetime
import os
def rotate_log_if_new_day(self):
now = datetime.now().date()
file_date = datetime.fromtimestamp(os.path.getmtime("manager.err")).date()
if file_date != now:
self.logger.debug("Date changed with no events - rotating the log...")
self._errhandler.doRollover()
rotate_log_if_new_day() is called during the manager’s update cycle (every 15 minutes) which guarantees the log will rotate within 15 minutes of midnight. If the log has already rotated (ie: an error was logged or it already did the manual rotation), it does nothing.
Originally I had a guard to prevent the os.path... call unless the date had changed, but timeit() shows a negligible speedup gained by caching the current date (1.85μs vs 5.7μs) to verify no action required.
I wrote a custom Handler, through which you can achieve the functionality you need.
import logging
import os
from custom_rotating_file_handler import CustomRotatingFileHandler
def custom_rotating_logger(log_fpath, when='H'):
os.makedirs(os.path.dirname(log_fpath), exist_ok=True)
file_handler = CustomRotatingFileHandler(log_fpath,
when=when,
backupCount=10,
encoding="utf-8")
formatter = logging.Formatter('%(asctime)s %(message)s')
file_handler.setFormatter(formatter)
logger = logging.getLogger('MyCustomLogger')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
return logger
# For test: log rotating at beginning of each minute.
logger = custom_rotating_logger('/tmp/tmp.log', when='M')
import time
for i in range(100):
time.sleep(1)
logger.debug('ddddd')
logger.info('iiiii')
logger.warning('wwwww')
Save the code above as testhandler.py and run the following command:
python3 testhandler.py && sudo find /tmp -name 'tmp.log.*'
You can see the output like this:
/tmp/tmp.log.202212142132
/tmp/tmp.log.202212142133
Content in the log file:
> head -n 3 /tmp/tmp.log.202212142133
2022-12-14 21:33:00,483 ddddd
2022-12-14 21:33:00,490 iiiii
2022-12-14 21:33:00,490 wwwww
EDIT:
Looks like other people are having a similar issue with TimedRotatingFileHandler.
Why doesn't my TimedRotatingFileHandler rotate at midnight?
Apparently, the logs don’t rotate unless there is some activity happening in the logs. Is there a way I can achieve the scheduled rotating functionality I want using the builtin Handlers without having to create a custom rotator?
ORIGINAL POST:
I’m using the TimedRotatingFileHandler of Python’s logging module to rotate my logs regularly.
I’ve specified the configurations in logging.conf for my app’s loggers and handlers as such:
[logger_worker]
level=DEBUG
propogate=0
qualname=worker
handlers=workerHandler
[handler_workerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=standardFormatter
args=("/var/log/app/worker.log","M",1,30,None,False,False)
- Note: for testing purposes I’ve configured the handler to rotate the logs on every minute, but ideally the logs will be rotated on a daily basis at midnight.
In my app, I am creating the logger like this:
logging.config.fileConfig("logging.conf")
log = logging.getLogger("worker")
log.debug('Hello world')
It is not working as I expected it to work:
- It is not rotating all the logs
- It is not rotating every minute as it is configured to do
Observe the ls -l output of the log directory:
-rw-r--r-- 1 root root 0 Apr 19 18:22 dpv.log
-rw-r----- 1 root root 5092 Apr 20 11:47 emperor.log
-rw-r--r-- 1 root root 88939 Apr 20 11:47 uwsgi.log
-rw-r--r-- 1 root root 494 Apr 20 11:46 worker.log
-rw-r--r-- 1 root root 45906 Apr 20 02:08 worker.log.2016-04-20_02-08
-rw-r--r-- 1 root root 494 Apr 20 11:34 worker.log.2016-04-20_11-34
-rw-r--r-- 1 root root 494 Apr 20 11:36 worker.log.2016-04-20_11-36
-rw-r--r-- 1 root root 494 Apr 20 11:44 worker.log.2016-04-20_11-44
What am I doing wrong? Is it possible to rotate the logs on a scheduled term even when nothing is being written to the logs?
Seems TimedRotatingFileHandler doesn’t have the functionality that you need.
You can try the following function for rotating the log files at the certain time.
def log_rollover(folder_path):
# folder_path: the absolute path to your log folder
today = str(datetime.date.today()).replace('-', '')
list_of_files = os.listdir(folder_path)
for log_file in list_of_files:
file_path = folder_path + '/' + log_file
if log_file.endswith('.log') and os.stat(file_path).st_size > 0:
new_file_path = file_path.split('.')[-2] + '-' + today + '.log'
subprocess.check_call(['cp', file_path, new_file_path])
subprocess.check_call(['gzip', new_file_path])
subprocess.check_call(['cp', '/dev/null', file_path])
Also you need to use a cron job to run the function at the certain time.
APScheduler can be a good option. For example if you want to run the cron job at 3 AM, you can use:
folderPath = '/var/log/app' # The path to your log folder
scheduler = BlockingScheduler()
scheduler.add_job(log_rollover, trigger='cron', args=[folderPath], hour=3, minute=0)
scheduler.start()
Don’t forget to set your timezone for APScheduler.
I had the same issue and came up with a different approach:
from datetime import datetime
import os
def rotate_log_if_new_day(self):
now = datetime.now().date()
file_date = datetime.fromtimestamp(os.path.getmtime("manager.err")).date()
if file_date != now:
self.logger.debug("Date changed with no events - rotating the log...")
self._errhandler.doRollover()
rotate_log_if_new_day() is called during the manager’s update cycle (every 15 minutes) which guarantees the log will rotate within 15 minutes of midnight. If the log has already rotated (ie: an error was logged or it already did the manual rotation), it does nothing.
Originally I had a guard to prevent the os.path... call unless the date had changed, but timeit() shows a negligible speedup gained by caching the current date (1.85μs vs 5.7μs) to verify no action required.
I wrote a custom Handler, through which you can achieve the functionality you need.
import logging
import os
from custom_rotating_file_handler import CustomRotatingFileHandler
def custom_rotating_logger(log_fpath, when='H'):
os.makedirs(os.path.dirname(log_fpath), exist_ok=True)
file_handler = CustomRotatingFileHandler(log_fpath,
when=when,
backupCount=10,
encoding="utf-8")
formatter = logging.Formatter('%(asctime)s %(message)s')
file_handler.setFormatter(formatter)
logger = logging.getLogger('MyCustomLogger')
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
return logger
# For test: log rotating at beginning of each minute.
logger = custom_rotating_logger('/tmp/tmp.log', when='M')
import time
for i in range(100):
time.sleep(1)
logger.debug('ddddd')
logger.info('iiiii')
logger.warning('wwwww')
Save the code above as testhandler.py and run the following command:
python3 testhandler.py && sudo find /tmp -name 'tmp.log.*'
You can see the output like this:
/tmp/tmp.log.202212142132
/tmp/tmp.log.202212142133
Content in the log file:
> head -n 3 /tmp/tmp.log.202212142133
2022-12-14 21:33:00,483 ddddd
2022-12-14 21:33:00,490 iiiii
2022-12-14 21:33:00,490 wwwww