Logging messages appear twice in console Python
Question:
I have found this answer to a seemingly similar issue, however (since I’m novice into Python) I am not sure how to implement this solution in my code (if it’s the same issue after all).
In my code I have the following section:
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
filename='C:\Tests\TRACE.log',
filemode='a')
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleFormatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(consoleFormatter)
logging.getLogger('').addHandler(console)
localLog = logging.getLogger('text')
The funny thing is that it used to work fine but at some moment it started writing these duplicate messages to console.
Could someone give me a direction here please?
Answers:
Typically duplicate log statements occur because there are two separate handlers attached that are directing your log statements to the same place. There are a couple of things worth trying to get to the root of the problem:
- Comment out the call to logging.basicConfig – if this eliminates the duplicate log statements then this means it is likely that you don’t need to manually configure the second log handler.
- If you are using an IDE, it may be worth putting a breakpoint onto a log statement, and stepping in using the debugger so you can introspect the state of pythons log setup to get a clearer picture of what different handlers are attached.
To make your logging easier to manage, it may be worth looking to move the configuration out of the code and into a configuration file – the Python document on logging configuration is a great place to start.
It seems that I have figured out the source of this problem.
The thing is that I used to get logger at module level. It looked pretty logical but there is a pitfall – Python logging module respects all created logger before you load the configuration from a file. So basically, when I was importing a module(which uses gets logger internally) to a main code(where I was calling for a logger as well) it resulted in streaming the logger data twice.
The possible solutions to this problem are:
- Do not get logger at the module level
- Set
disable_existing_loggers
to False. Added since Python 2.7
I had the same situation with my logger configuration, the fix that i used was:
class Logger:
@staticmethod
def setup(name, file_name):
log_file = <path to save the file>
log_file_max_size = 1024 * 1024 * 20 # megabytes
log_num_backups = 3
log_format = "%(asctime)s [%(levelname)s]: %(filename)s(%(funcName)s:%(lineno)s) >> %(message)s"
log_filemode = "w" # w: overwrite; a: append
logging.basicConfig(filename=log_file, format=log_format, filemode=log_filemode, level=logging.DEBUG)
rotate_file = logging.handlers.RotatingFileHandler(
log_file, maxBytes=log_file_max_size, backupCount=log_num_backups
)
logger = logging.getLogger(name)
# Console output line.
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
log_formatter = logging.Formatter(log_format)
console_handler.setFormatter(log_formatter)
logger.handlers = rotate_file, console_handler
return logger
Here is some solution if you want to create logger in separate file. I used information from @(Joshua Briceño Z) answer, however I had some issues with saving logs to a file.
Basically I dropped the basicConfig, and to console_handler I added propagate=0 to skip printing twice.
Here I have two files:
- config.py – where is a Logger Class that setups the logger;
- program.py – where I call a Logger Class from config and tell to which file save the logs.
config.py:
import logging
class Logger:
"""
Sets the logger.
Attributes:
:file_name: set a filename in which logs will be saved.
:name: set the name of specific logger.
:return: function
"""
@staticmethod
def setup(name: str, file_name: str):
log_file = file_name
log_format = "%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s"
log_level = logging.INFO
logger = logging.getLogger(name)
# File output line
logger.setLevel(logging.INFO)
formatter = logging.Formatter(log_format)
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
# Console output line
console_handler = logging.StreamHandler()
console_handler.setLevel(log_level)
log_formatter = logging.Formatter(log_format)
console_handler.setFormatter(log_formatter)
logger.addHandler(console_handler)
logger.propagate = 0
return logger
program.py
import config
logger_1 = config.Logger.setup(name='program', file_name='program.log')
NOTE: Tested on python 3.8.10.
I have found this answer to a seemingly similar issue, however (since I’m novice into Python) I am not sure how to implement this solution in my code (if it’s the same issue after all).
In my code I have the following section:
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
filename='C:\Tests\TRACE.log',
filemode='a')
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleFormatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(consoleFormatter)
logging.getLogger('').addHandler(console)
localLog = logging.getLogger('text')
The funny thing is that it used to work fine but at some moment it started writing these duplicate messages to console.
Could someone give me a direction here please?
Typically duplicate log statements occur because there are two separate handlers attached that are directing your log statements to the same place. There are a couple of things worth trying to get to the root of the problem:
- Comment out the call to logging.basicConfig – if this eliminates the duplicate log statements then this means it is likely that you don’t need to manually configure the second log handler.
- If you are using an IDE, it may be worth putting a breakpoint onto a log statement, and stepping in using the debugger so you can introspect the state of pythons log setup to get a clearer picture of what different handlers are attached.
To make your logging easier to manage, it may be worth looking to move the configuration out of the code and into a configuration file – the Python document on logging configuration is a great place to start.
It seems that I have figured out the source of this problem.
The thing is that I used to get logger at module level. It looked pretty logical but there is a pitfall – Python logging module respects all created logger before you load the configuration from a file. So basically, when I was importing a module(which uses gets logger internally) to a main code(where I was calling for a logger as well) it resulted in streaming the logger data twice.
The possible solutions to this problem are:
- Do not get logger at the module level
- Set
disable_existing_loggers
to False. Added since Python 2.7
I had the same situation with my logger configuration, the fix that i used was:
class Logger:
@staticmethod
def setup(name, file_name):
log_file = <path to save the file>
log_file_max_size = 1024 * 1024 * 20 # megabytes
log_num_backups = 3
log_format = "%(asctime)s [%(levelname)s]: %(filename)s(%(funcName)s:%(lineno)s) >> %(message)s"
log_filemode = "w" # w: overwrite; a: append
logging.basicConfig(filename=log_file, format=log_format, filemode=log_filemode, level=logging.DEBUG)
rotate_file = logging.handlers.RotatingFileHandler(
log_file, maxBytes=log_file_max_size, backupCount=log_num_backups
)
logger = logging.getLogger(name)
# Console output line.
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
log_formatter = logging.Formatter(log_format)
console_handler.setFormatter(log_formatter)
logger.handlers = rotate_file, console_handler
return logger
Here is some solution if you want to create logger in separate file. I used information from @(Joshua Briceño Z) answer, however I had some issues with saving logs to a file.
Basically I dropped the basicConfig, and to console_handler I added propagate=0 to skip printing twice.
Here I have two files:
- config.py – where is a Logger Class that setups the logger;
- program.py – where I call a Logger Class from config and tell to which file save the logs.
config.py:
import logging
class Logger:
"""
Sets the logger.
Attributes:
:file_name: set a filename in which logs will be saved.
:name: set the name of specific logger.
:return: function
"""
@staticmethod
def setup(name: str, file_name: str):
log_file = file_name
log_format = "%(asctime)s - %(name)-12s: %(levelname)-8s %(message)s"
log_level = logging.INFO
logger = logging.getLogger(name)
# File output line
logger.setLevel(logging.INFO)
formatter = logging.Formatter(log_format)
file_handler = logging.FileHandler(log_file)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
# Console output line
console_handler = logging.StreamHandler()
console_handler.setLevel(log_level)
log_formatter = logging.Formatter(log_format)
console_handler.setFormatter(log_formatter)
logger.addHandler(console_handler)
logger.propagate = 0
return logger
program.py
import config
logger_1 = config.Logger.setup(name='program', file_name='program.log')
NOTE: Tested on python 3.8.10.