Python Logging – Set Date as Filename
Question:
I am working on implementing logging within my Python project and have hit a bit of a snag. I am trying to set up my logging such that the Handlers, and Formatters are all organized into a configuration file. What I am trying to do at the moment is to set up my fileHandler
such that it will create a log file that looks something like this: YYYY_MM_DD.log
obviously with the Y’s representing the year, M’s representing the month, and D’s representing the day.
This is what I have attempted with my config file:
[loggers]
keys=root,MainLogger
[handlers]
keys=fileHandler, consoleHandler
[formatters]
keys=logFormatter, consoleFormatter
[logger_root]
level=DEBUG
handlers=fileHandler
[logger_MainLogger]
level=DEBUG
handlers=fileHandler, consoleHandler
qualname=MainLogger
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=logFormatter
args=(datetime.now().strftime('%Y_%m_%d.log'), 'a')
[formatter_logFormatter]
format=%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(fillname)s-%(funcName)s-%(lineno)04d | %message)s
The file I am using to test the configuration is pretty simple:
import logging
import logging.config
logging.config.fileConfig('logging.conf')
logger = logging.getLogger('MainLogger')
logger.debug("TEST")
The specific error I am getting at the moment is:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: "%Y_%m_%d.log'), 'a')"
I’ve tried changing the %Y
, %m
, and %d
as the error says, but that doesn’t fix the problem. How do I go about setting up the config file so that my log files look the way I want them to?
I should note when I change the filename to test.log
everything worked fine, so this is the only error I seem to be having with this.
Answers:
This uses content from your config file, but does not access the file directly. You create your own filehandler and then add it to the logger.
import logging
from datetime import datetime
# Create logger.
logger = logging.getLogger('MainLogger')
logger.setLevel(logging.DEBUG)
# Create filehandler with desired filename.
fh = logging.FileHandler('{}.log'.format(datetime.now().strftime('%Y_%m_%d')))
fh.setLevel(logging.DEBUG)
log_formatter = logging.Formatter('%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s')
fh.setFormatter(log_formatter)
# Add filehandler to logger.
logger.addHandler(fh)
Note that a
(append) is the default mode
parameter for a FileHandler.
Perhaps you can use Python’s TimedRotatingFileHandler
instead. You can set the interval to create a new log file every day with the date as the suffix.
Documentation–
- Python 2: https://docs.python.org/2/library/logging.handlers.html#timedrotatingfilehandler
- Python 3: https://docs.python.org/3/library/logging.handlers.html#timedrotatingfilehandler
Note that the current day’s log file won’t have a date. This file handler only adds the date suffix when a new day starts.
Also, the suffix it uses is “%Y-%m-%d”, which is a little different than what you want. But there’s a SO question here about how you can alter that.
Maybe try changing the name after you’ve loaded the config file:
from datetime inport datetime
logging.config.fileConfig('logging.conf')
logging.basicConfig(filename = datetime.now().strftime('%Y_%m_%d.log'))
You can’t use datetime
in a config file, as it doesn’t know what it means. You can however add the Filehandler
in the python file itself:
import logging.config
from datetime import datetime
logging.config.fileConfig('aaa.conf')
logger = logging.getLogger('MainLogger')
fh = logging.FileHandler('{:%Y-%m-%d}.log'.format(datetime.now()))
formatter = logging.Formatter('%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug("TEST")
This way you can set the date as the file name in the handler.
This is the config file, note that you had a typo in the last formatter, you put fillname
instead of filename
and you forgot (
in message
.
[loggers]
keys=root,MainLogger
[handlers]
keys=consoleHandler
[formatters]
keys=consoleFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_MainLogger]
level=DEBUG
handlers=consoleHandler
qualname=MainLogger
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(filename)s-%(funcName)s-%(lineno)04d | %(message)s
This Should work just fine.
This worked for me.
Update this,
args=(datetime.now().strftime('%Y_%m_%d.log'), 'a')
with this,
args=(__import__("datetime").datetime.now().strftime('%Y_%m_%d.log'), 'a')
Reference (Example no 3): http://python-reference.readthedocs.io/en/latest/docs/functions/eval.html
This also works
from datetime import datetime
log_file = str(datetime.utcnow().strftime('%m_%d_%Y_%I_%M_%S')) + '.log'
logging.basicConfig(filename=log_file, format='%(levelname)s | %(asctime)s | %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', level=logging.DEBUG)
Logfile name:04_30_2018_10_03_01.log
Using double ‘%’-characters in the format string in combination with the approach suggested by Abhishek led to a working solution in my case (Python 3.5):
The filehandler in the config file should then look similar to this one:
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=defaultFormatter
args=(__import__("datetime").datetime.now().strftime('/your_path/your_file_name_%%Y-%%m-%%d_%%H-%%M-%%S.log'), 'a')
This post is old, but for anyone still struggling with any such issue of dynamic naming of filenames from logging.conf
while calling logging.config.fileConfig() you can pass the variables and use them in the logging configuration file.
Eg:
logging.conf
args=('my_log_%(date)s.log','w')
python.py
import logging.config
logging.config.fileConfig('logging.conf', defaults={'date':datetime.now()})
You might wanna format the date while using it here.
In the same fashion, any number of variables could be used.
My solution for my python product is to keep the configuration file with a path to a fixed log file such as:
[handler_fileHandler] <br>
class=FileHandler <br>
level=DEBUG <br>
formatter=defaultFormatter <br>
args=('../projectnamespace/data/logs/logfile.log',)
and for each day I will move the log file to older folder logs so by doing that I avoid having a huge log files for later reading….
log_path = str((Path(__file__).parent / 'data' / 'logs' / 'logfile.log').absolute())
log_file_date_created = None
if platform.system() == 'Windows':
log_file_date_created = datetime.fromtimestamp(os.path.getctime(log_path))
else:
stat = os.stat(log_path)
try:
log_file_date_created = stat.st_birthtime
except AttributeError:
# running in Linux. No easy way to get creation dates here,
# we get when its content was last modified.
return stat.st_mtime
if log_file_date_created != None and abs(datetime.today().day - log_file_date_created.day) >= 1:
file_date = str(log_file_date_created.strftime("%m_%d_%Y")) + '_logfile.log'
log_path_new_name = str((Path(__file__).parent / 'data' / 'logs' / 'older' / file_date ).absolute())
os.replace(log_path, log_path_new_name)
I wanted to keep all of my logging configurable through the config file and wasn’t satisfied with the answers here manually adding the handler through the code. The solution I came up with is to create a handler class inheriting FileHandler which appends the date to the filename parameter before calling the FileHandler constructor:
import os
from logging import FileHandler
from datetime import datetime
class TimestampedFileHandler(FileHandler):
def __init__(self, filename, mode='a', encoding=None, delay=False):
filename, extension = os.path.splitext(filename)
filename = f"{filename}_{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}{extension}"
FileHandler.__init__(self, filename, mode, encoding, delay)
You can then use the TimestampedFileHandler in your config file and set its level, formatter, and parameters together with the other handlers.
I created class called LoggingService and use it’s in other python file
import logging
from logging.handlers import RotatingFileHandler
from datetime import datetime
class LoggingService:
def namer(name):
now = datetime.now()
date_time = now.strftime("%Y-%m-%d")
return name[:-1] + date_time +"."+name[-1:]
LOG_FORMAT = "%(levelname)s %(asctime)s - %(name)s - %(message)s"
log_file = "your inital log file location"
rotate_handler = RotatingFileHandler(
filename=log_file,
mode='a',
maxBytes=256,
backupCount=8,
encoding=None,
delay=False
)
logging.basicConfig(
level=logging.DEBUG,
format=LOG_FORMAT,
datefmt='%m/%d/%Y %I:%M:%S %p',
handlers=[rotate_handler]
)
rotate_handler.namer = namer
@staticmethod
def get_logger(name):
return logging.getLogger(name)
My solution is to use f strings.
Place f
into the ‘filename’ parameter in your config and add {date.today()}
or {datetime.now()}
.
import logging
from datetime import date, datetime
logging.basicConfig(filename=f"C:\Users\...\my_log{date.today()}.log",
encoding='utf-8', format='%(asctime)s - %(message)s', level=logging.INFO)
I am working on implementing logging within my Python project and have hit a bit of a snag. I am trying to set up my logging such that the Handlers, and Formatters are all organized into a configuration file. What I am trying to do at the moment is to set up my fileHandler
such that it will create a log file that looks something like this: YYYY_MM_DD.log
obviously with the Y’s representing the year, M’s representing the month, and D’s representing the day.
This is what I have attempted with my config file:
[loggers]
keys=root,MainLogger
[handlers]
keys=fileHandler, consoleHandler
[formatters]
keys=logFormatter, consoleFormatter
[logger_root]
level=DEBUG
handlers=fileHandler
[logger_MainLogger]
level=DEBUG
handlers=fileHandler, consoleHandler
qualname=MainLogger
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=logFormatter
args=(datetime.now().strftime('%Y_%m_%d.log'), 'a')
[formatter_logFormatter]
format=%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(fillname)s-%(funcName)s-%(lineno)04d | %message)s
The file I am using to test the configuration is pretty simple:
import logging
import logging.config
logging.config.fileConfig('logging.conf')
logger = logging.getLogger('MainLogger')
logger.debug("TEST")
The specific error I am getting at the moment is:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: "%Y_%m_%d.log'), 'a')"
I’ve tried changing the %Y
, %m
, and %d
as the error says, but that doesn’t fix the problem. How do I go about setting up the config file so that my log files look the way I want them to?
I should note when I change the filename to test.log
everything worked fine, so this is the only error I seem to be having with this.
This uses content from your config file, but does not access the file directly. You create your own filehandler and then add it to the logger.
import logging
from datetime import datetime
# Create logger.
logger = logging.getLogger('MainLogger')
logger.setLevel(logging.DEBUG)
# Create filehandler with desired filename.
fh = logging.FileHandler('{}.log'.format(datetime.now().strftime('%Y_%m_%d')))
fh.setLevel(logging.DEBUG)
log_formatter = logging.Formatter('%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s')
fh.setFormatter(log_formatter)
# Add filehandler to logger.
logger.addHandler(fh)
Note that a
(append) is the default mode
parameter for a FileHandler.
Perhaps you can use Python’s TimedRotatingFileHandler
instead. You can set the interval to create a new log file every day with the date as the suffix.
Documentation–
- Python 2: https://docs.python.org/2/library/logging.handlers.html#timedrotatingfilehandler
- Python 3: https://docs.python.org/3/library/logging.handlers.html#timedrotatingfilehandler
Note that the current day’s log file won’t have a date. This file handler only adds the date suffix when a new day starts.
Also, the suffix it uses is “%Y-%m-%d”, which is a little different than what you want. But there’s a SO question here about how you can alter that.
Maybe try changing the name after you’ve loaded the config file:
from datetime inport datetime
logging.config.fileConfig('logging.conf')
logging.basicConfig(filename = datetime.now().strftime('%Y_%m_%d.log'))
You can’t use datetime
in a config file, as it doesn’t know what it means. You can however add the Filehandler
in the python file itself:
import logging.config
from datetime import datetime
logging.config.fileConfig('aaa.conf')
logger = logging.getLogger('MainLogger')
fh = logging.FileHandler('{:%Y-%m-%d}.log'.format(datetime.now()))
formatter = logging.Formatter('%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.debug("TEST")
This way you can set the date as the file name in the handler.
This is the config file, note that you had a typo in the last formatter, you put fillname
instead of filename
and you forgot (
in message
.
[loggers]
keys=root,MainLogger
[handlers]
keys=consoleHandler
[formatters]
keys=consoleFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_MainLogger]
level=DEBUG
handlers=consoleHandler
qualname=MainLogger
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(filename)s-%(funcName)s-%(lineno)04d | %(message)s
This Should work just fine.
This worked for me.
Update this,
args=(datetime.now().strftime('%Y_%m_%d.log'), 'a')
with this,
args=(__import__("datetime").datetime.now().strftime('%Y_%m_%d.log'), 'a')
Reference (Example no 3): http://python-reference.readthedocs.io/en/latest/docs/functions/eval.html
This also works
from datetime import datetime
log_file = str(datetime.utcnow().strftime('%m_%d_%Y_%I_%M_%S')) + '.log'
logging.basicConfig(filename=log_file, format='%(levelname)s | %(asctime)s | %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', level=logging.DEBUG)
Logfile name:04_30_2018_10_03_01.log
Using double ‘%’-characters in the format string in combination with the approach suggested by Abhishek led to a working solution in my case (Python 3.5):
The filehandler in the config file should then look similar to this one:
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=defaultFormatter
args=(__import__("datetime").datetime.now().strftime('/your_path/your_file_name_%%Y-%%m-%%d_%%H-%%M-%%S.log'), 'a')
This post is old, but for anyone still struggling with any such issue of dynamic naming of filenames from logging.conf
while calling logging.config.fileConfig() you can pass the variables and use them in the logging configuration file.
Eg:
logging.conf
args=('my_log_%(date)s.log','w')
python.py
import logging.config
logging.config.fileConfig('logging.conf', defaults={'date':datetime.now()})
You might wanna format the date while using it here.
In the same fashion, any number of variables could be used.
My solution for my python product is to keep the configuration file with a path to a fixed log file such as:
[handler_fileHandler] <br>
class=FileHandler <br>
level=DEBUG <br>
formatter=defaultFormatter <br>
args=('../projectnamespace/data/logs/logfile.log',)
and for each day I will move the log file to older folder logs so by doing that I avoid having a huge log files for later reading….
log_path = str((Path(__file__).parent / 'data' / 'logs' / 'logfile.log').absolute())
log_file_date_created = None
if platform.system() == 'Windows':
log_file_date_created = datetime.fromtimestamp(os.path.getctime(log_path))
else:
stat = os.stat(log_path)
try:
log_file_date_created = stat.st_birthtime
except AttributeError:
# running in Linux. No easy way to get creation dates here,
# we get when its content was last modified.
return stat.st_mtime
if log_file_date_created != None and abs(datetime.today().day - log_file_date_created.day) >= 1:
file_date = str(log_file_date_created.strftime("%m_%d_%Y")) + '_logfile.log'
log_path_new_name = str((Path(__file__).parent / 'data' / 'logs' / 'older' / file_date ).absolute())
os.replace(log_path, log_path_new_name)
I wanted to keep all of my logging configurable through the config file and wasn’t satisfied with the answers here manually adding the handler through the code. The solution I came up with is to create a handler class inheriting FileHandler which appends the date to the filename parameter before calling the FileHandler constructor:
import os
from logging import FileHandler
from datetime import datetime
class TimestampedFileHandler(FileHandler):
def __init__(self, filename, mode='a', encoding=None, delay=False):
filename, extension = os.path.splitext(filename)
filename = f"{filename}_{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}{extension}"
FileHandler.__init__(self, filename, mode, encoding, delay)
You can then use the TimestampedFileHandler in your config file and set its level, formatter, and parameters together with the other handlers.
I created class called LoggingService and use it’s in other python file
import logging
from logging.handlers import RotatingFileHandler
from datetime import datetime
class LoggingService:
def namer(name):
now = datetime.now()
date_time = now.strftime("%Y-%m-%d")
return name[:-1] + date_time +"."+name[-1:]
LOG_FORMAT = "%(levelname)s %(asctime)s - %(name)s - %(message)s"
log_file = "your inital log file location"
rotate_handler = RotatingFileHandler(
filename=log_file,
mode='a',
maxBytes=256,
backupCount=8,
encoding=None,
delay=False
)
logging.basicConfig(
level=logging.DEBUG,
format=LOG_FORMAT,
datefmt='%m/%d/%Y %I:%M:%S %p',
handlers=[rotate_handler]
)
rotate_handler.namer = namer
@staticmethod
def get_logger(name):
return logging.getLogger(name)
My solution is to use f strings.
Place f
into the ‘filename’ parameter in your config and add {date.today()}
or {datetime.now()}
.
import logging
from datetime import date, datetime
logging.basicConfig(filename=f"C:\Users\...\my_log{date.today()}.log",
encoding='utf-8', format='%(asctime)s - %(message)s', level=logging.INFO)