Duplicate log output when using Python logging module
Question:
I am using python logger. The following is my code:
import os
import time
import datetime
import logging
class Logger :
def myLogger(self):
logger = logging.getLogger('ProvisioningPython')
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('/root/credentials/Logs/ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
The problem I have is that I get multiple entries in the log file for each logger.info
call. How can I solve this?
Answers:
You are calling Logger.myLogger()
more than once. Store the logger instance it returns somewhere and reuse that.
Also be advised that if you log before any handler is added, a default StreamHandler(sys.stderr)
will be created.
Your logger should work as singleton. You shouldn’t create it more than once.
Here is example how it might look:
import os
import time
import datetime
import logging
class Logger :
logger = None
def myLogger(self):
if None == self.logger:
self.logger=logging.getLogger('ProvisioningPython')
self.logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
self.logger.addHandler(handler)
return self.logger
s = Logger()
m = s.myLogger()
m2 = s.myLogger()
m.info("Info1")
m2.info("info2")
The logging.getLogger()
is returns the same instance for a given name. (Documentation)
The problem is that every time you call myLogger()
, it’s adding another handler to the instance, which causes the duplicate logs.
Perhaps something like this?
import os
import time
import datetime
import logging
loggers = {}
def myLogger(name):
global loggers
if loggers.get(name):
return loggers.get(name)
else:
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler = logging.FileHandler(
'/root/credentials/Logs/ProvisioningPython'
+ now.strftime("%Y-%m-%d")
+ '.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
loggers[name] = logger
return logger
import datetime
import logging
class Logger :
def myLogger(self):
logger=logging.getLogger('ProvisioningPython')
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('/root/credentials/Logs/ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
made the trick for me
using python 2.7
The implementation of logger is already a singleton.
Multiple calls to logging.getLogger(‘someLogger’) return a reference
to the same logger object. This is true not only within the same
module, but also across modules as long as it is in the same Python
interpreter process. It is true for references to the same object;
additionally, application code can define and configure a parent
logger in one module and create (but not configure) a child logger in
a separate module, and all logger calls to the child will pass up to
the parent. Here is a main module
Source- Using logging in multiple modules
So the way you should utilize this is –
Let’s suppose we have created and configured a logger called ‘main_logger’ in the main module (which simply configures the logger, doesn’t return anything).
# get the logger instance
logger = logging.getLogger("main_logger")
# configuration follows
...
Now in a sub-module, if we create a child logger following the naming hierarchy ‘main_logger.sub_module_logger’, we don’t need to configure it in the sub-module. Just creation of the logger following the naming hierarchy is sufficient.
# get the logger instance
logger = logging.getLogger("main_logger.sub_module_logger")
# no configuration needed
# it inherits the configuration from the parent logger
...
And it won’t add duplicate handler as well.
See this question for a little more verbose answer.
Double (or triple or ..- based on number of reloads) logger output may also happen when you reload your module via importlib.reload
(for the same reason as explained in accepted answer). I am adding this answer just for a future reference as it took me a while to figure out why my output is dupli(triple)cated.
One simple workaround is like
logger.handlers[:] = [handler]
This way you avoid appending new handler to the underlying list “handlers”.
from logging.handlers import RotatingFileHandler
import logging
import datetime
# stores all the existing loggers
loggers = {}
def get_logger(name):
# if a logger exists, return that logger, else create a new one
global loggers
if name in loggers.keys():
return loggers[name]
else:
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler = logging.FileHandler(
'path_of_your_log_file'
+ now.strftime("%Y-%m-%d")
+ '.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
loggers.update(dict(name=logger))
return logger
You are able to get list of all handlers for the particular logger, so you can do something like this
logger = logging.getLogger(logger_name)
handler_installed = False
for handler in logger:
# Here your condition to check for handler presence
if isinstance(handler, logging.FileHandler) and handler.baseFilename == log_filename:
handler_installed = True
break
if not handler_installed:
logger.addHandler(your_handler)
In the example above we check if the handler for a file specified is already hooked to the logger, but having access to the list of all handlers gives you an ability to decide on which criteria you should add another handler or not.
Had this problem today. Since my functions were @staticmethod the above suggestions were resolved with random().
Looking something like:
import random
logger = logging.getLogger('ProvisioningPython.{}'.format(random.random()))
Since Python 3.2 you can just check if handlers are already present and if so, clear them before adding new handlers. This is pretty convenient when debugging and the code includes your logger initialization
if (logger.hasHandlers()):
logger.handlers.clear()
logger.addHandler(handler)
This is an addition to @rm957377’s answer but with an explanation of why this is happening. When you run a lambda function in AWS, they call your function from within a wrapping instance that stays alive for multiple calls. Meaning, if you call addHandler()
within the code of your function, it will continue to add duplicate handlers to the logging singleton every time the function runs. The logging singleton persists through multiple calls of you lambda function.
To solve this you can clear your handlers before you set them via:
logging.getLogger().handlers.clear()
logging.getLogger().addHandler(...)
Bottom line for most cases when this happens, one only needs to call logger.getLogger() only once per module. If you have multiple classes like I did, I could call it like so:
LOGGER = logger.getLogger(__name__)
class MyClass1:
log = LOGGER
def __init__(self):
self.log.debug('class 1 initialized')
class MyClass2:
log = LOGGER
def __init__(self):
self.log.debug('class 2 initialized')
Both then will have their own full package name and method where logged.
I already used the logger
as a Singleton and checked if not len(logger.handlers)
, but still got duplicates: It was the formatted output, followed by the unformatted.
Solution in my case:
logger.propagate = False
Credits to this answer and the docs.
I have 3 handlers in one logger
StreamHandler setLevel(args.logging_level)
logging.FileHandler(logging.ERROR)
RotatingFileHandler(args.logging_level)
logger.setLevel(args.logging_level)
I had my code using
logger = logging.getLogger('same_name_everywhere')
resulting duplicated lines and duplicated handlers like this, 2 Stream Handlers, 3 Rotating FileHanders
While 1 Stream Handler + 2 Rotating FileHanders(1 for errlog, 1 for generic log)
This is done by
logger.warn(logger.handlers)
cli_normalize_string: WARNING [<StreamHandler <stderr> (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>, <StreamHandler <stderr> (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.log (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>]
After I changed to
# The name is now become change.cli_normalize_string or change.normalize_string
logger = logger.getLogger(__name__)
in every modules, issue resolved, no duplicated lines, 1 StreamHeader, 1 FileHandler for err logging, 1 RotatingFileHandler for generic logging
2020-11-02 21:26:05,856 cli_normalize_string INFO [<StreamHandler <stderr> (DEBUG)>, <FileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.log (DEBUG)>]
The details is in this document
https://docs.python.org/3/library/logging.html
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name). Multiple calls to getLogger() with the same name will always return a reference to the same Logger object."
The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo,
loggers with names of
foo.bar
foo.bar.baz
foo.bam
are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise
your loggers on a per-module basis using the recommended construction
logging.getLogger(__name__).
That’s because in a module,
__name__
is the module’s name in the Python package namespace.
When we use getLogger() without any argument, it returns RootLogger.
So if you invoke getLogger() at multiple places and add log handler, it will add those log handlers to the RootLogger (If you don’t add log handler explicitly, it will add StreamHandler automatically). Thus, when you will try to log messages, it will log messages using all the handlers added to RootLogger. This is the cause of duplicate logs.
You can avoid this by just providing a different logger name when you invoke getLogger(). Something like
logger1 = logging.getLogger("loggera")
logger2 = logging.getLogger("loggerb")
This is something that worked for me.
The problem is the number handlers, if you have multiple handlers you have multiples logs so you just have to check before add:
if not logger.handlers:
logger.addHandler(handler)
I am using python logger. The following is my code:
import os
import time
import datetime
import logging
class Logger :
def myLogger(self):
logger = logging.getLogger('ProvisioningPython')
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('/root/credentials/Logs/ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
The problem I have is that I get multiple entries in the log file for each logger.info
call. How can I solve this?
You are calling Logger.myLogger()
more than once. Store the logger instance it returns somewhere and reuse that.
Also be advised that if you log before any handler is added, a default StreamHandler(sys.stderr)
will be created.
Your logger should work as singleton. You shouldn’t create it more than once.
Here is example how it might look:
import os
import time
import datetime
import logging
class Logger :
logger = None
def myLogger(self):
if None == self.logger:
self.logger=logging.getLogger('ProvisioningPython')
self.logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
self.logger.addHandler(handler)
return self.logger
s = Logger()
m = s.myLogger()
m2 = s.myLogger()
m.info("Info1")
m2.info("info2")
The logging.getLogger()
is returns the same instance for a given name. (Documentation)
The problem is that every time you call myLogger()
, it’s adding another handler to the instance, which causes the duplicate logs.
Perhaps something like this?
import os
import time
import datetime
import logging
loggers = {}
def myLogger(name):
global loggers
if loggers.get(name):
return loggers.get(name)
else:
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler = logging.FileHandler(
'/root/credentials/Logs/ProvisioningPython'
+ now.strftime("%Y-%m-%d")
+ '.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
loggers[name] = logger
return logger
import datetime
import logging
class Logger :
def myLogger(self):
logger=logging.getLogger('ProvisioningPython')
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler=logging.FileHandler('/root/credentials/Logs/ProvisioningPython'+ now.strftime("%Y-%m-%d") +'.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
return logger
made the trick for me
using python 2.7
The implementation of logger is already a singleton.
Multiple calls to logging.getLogger(‘someLogger’) return a reference
to the same logger object. This is true not only within the same
module, but also across modules as long as it is in the same Python
interpreter process. It is true for references to the same object;
additionally, application code can define and configure a parent
logger in one module and create (but not configure) a child logger in
a separate module, and all logger calls to the child will pass up to
the parent. Here is a main module
Source- Using logging in multiple modules
So the way you should utilize this is –
Let’s suppose we have created and configured a logger called ‘main_logger’ in the main module (which simply configures the logger, doesn’t return anything).
# get the logger instance
logger = logging.getLogger("main_logger")
# configuration follows
...
Now in a sub-module, if we create a child logger following the naming hierarchy ‘main_logger.sub_module_logger’, we don’t need to configure it in the sub-module. Just creation of the logger following the naming hierarchy is sufficient.
# get the logger instance
logger = logging.getLogger("main_logger.sub_module_logger")
# no configuration needed
# it inherits the configuration from the parent logger
...
And it won’t add duplicate handler as well.
See this question for a little more verbose answer.
Double (or triple or ..- based on number of reloads) logger output may also happen when you reload your module via importlib.reload
(for the same reason as explained in accepted answer). I am adding this answer just for a future reference as it took me a while to figure out why my output is dupli(triple)cated.
One simple workaround is like
logger.handlers[:] = [handler]
This way you avoid appending new handler to the underlying list “handlers”.
from logging.handlers import RotatingFileHandler
import logging
import datetime
# stores all the existing loggers
loggers = {}
def get_logger(name):
# if a logger exists, return that logger, else create a new one
global loggers
if name in loggers.keys():
return loggers[name]
else:
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
now = datetime.datetime.now()
handler = logging.FileHandler(
'path_of_your_log_file'
+ now.strftime("%Y-%m-%d")
+ '.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
loggers.update(dict(name=logger))
return logger
You are able to get list of all handlers for the particular logger, so you can do something like this
logger = logging.getLogger(logger_name)
handler_installed = False
for handler in logger:
# Here your condition to check for handler presence
if isinstance(handler, logging.FileHandler) and handler.baseFilename == log_filename:
handler_installed = True
break
if not handler_installed:
logger.addHandler(your_handler)
In the example above we check if the handler for a file specified is already hooked to the logger, but having access to the list of all handlers gives you an ability to decide on which criteria you should add another handler or not.
Had this problem today. Since my functions were @staticmethod the above suggestions were resolved with random().
Looking something like:
import random
logger = logging.getLogger('ProvisioningPython.{}'.format(random.random()))
Since Python 3.2 you can just check if handlers are already present and if so, clear them before adding new handlers. This is pretty convenient when debugging and the code includes your logger initialization
if (logger.hasHandlers()):
logger.handlers.clear()
logger.addHandler(handler)
This is an addition to @rm957377’s answer but with an explanation of why this is happening. When you run a lambda function in AWS, they call your function from within a wrapping instance that stays alive for multiple calls. Meaning, if you call addHandler()
within the code of your function, it will continue to add duplicate handlers to the logging singleton every time the function runs. The logging singleton persists through multiple calls of you lambda function.
To solve this you can clear your handlers before you set them via:
logging.getLogger().handlers.clear()
logging.getLogger().addHandler(...)
Bottom line for most cases when this happens, one only needs to call logger.getLogger() only once per module. If you have multiple classes like I did, I could call it like so:
LOGGER = logger.getLogger(__name__)
class MyClass1:
log = LOGGER
def __init__(self):
self.log.debug('class 1 initialized')
class MyClass2:
log = LOGGER
def __init__(self):
self.log.debug('class 2 initialized')
Both then will have their own full package name and method where logged.
I already used the logger
as a Singleton and checked if not len(logger.handlers)
, but still got duplicates: It was the formatted output, followed by the unformatted.
Solution in my case:
logger.propagate = False
Credits to this answer and the docs.
I have 3 handlers in one logger
StreamHandler setLevel(args.logging_level)
logging.FileHandler(logging.ERROR)
RotatingFileHandler(args.logging_level)
logger.setLevel(args.logging_level)
I had my code using
logger = logging.getLogger('same_name_everywhere')
resulting duplicated lines and duplicated handlers like this, 2 Stream Handlers, 3 Rotating FileHanders
While 1 Stream Handler + 2 Rotating FileHanders(1 for errlog, 1 for generic log)
This is done by
logger.warn(logger.handlers)
cli_normalize_string: WARNING [<StreamHandler <stderr> (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>, <StreamHandler <stderr> (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.log (DEBUG)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>]
After I changed to
# The name is now become change.cli_normalize_string or change.normalize_string
logger = logger.getLogger(__name__)
in every modules, issue resolved, no duplicated lines, 1 StreamHeader, 1 FileHandler for err logging, 1 RotatingFileHandler for generic logging
2020-11-02 21:26:05,856 cli_normalize_string INFO [<StreamHandler <stderr> (DEBUG)>, <FileHandler /tmp/cli.normalize_string.py.2020-11-02.user.errlog (ERROR)>, <RotatingFileHandler /tmp/cli.normalize_string.py.2020-11-02.user.log (DEBUG)>]
The details is in this document
https://docs.python.org/3/library/logging.html
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name). Multiple calls to getLogger() with the same name will always return a reference to the same Logger object."
The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo,
loggers with names of
foo.bar
foo.bar.baz
foo.bam
are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise
your loggers on a per-module basis using the recommended construction
logging.getLogger(__name__).
That’s because in a module,
__name__
is the module’s name in the Python package namespace.
When we use getLogger() without any argument, it returns RootLogger.
So if you invoke getLogger() at multiple places and add log handler, it will add those log handlers to the RootLogger (If you don’t add log handler explicitly, it will add StreamHandler automatically). Thus, when you will try to log messages, it will log messages using all the handlers added to RootLogger. This is the cause of duplicate logs.
You can avoid this by just providing a different logger name when you invoke getLogger(). Something like
logger1 = logging.getLogger("loggera")
logger2 = logging.getLogger("loggerb")
This is something that worked for me.
The problem is the number handlers, if you have multiple handlers you have multiples logs so you just have to check before add:
if not logger.handlers:
logger.addHandler(handler)