What is the point of setLevel in a python logging handler?
Question:
Let’s say I have the following code:
import logging
import logging.handlers
a = logging.getLogger('myapp')
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
# The effective log level is still logging.WARN
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
I expect that setting logging.DEBUG on the handler would cause debug-level messages to be written to the log file. However, this prints 30 for the effective level (equal to logging.WARNING, the default), and only logs the warn message to the log file, not the debug message.
It appears that the handler’s log level is being dropped on the floor, e.g. it’s silently ignored. Which makes me wonder, why have setLevel on the handler at all?
Answers:
It allows finer control. By default the root logger has WARNING level set, this means that it wont print messages with lower level(no matter how the handlers’ levels are set!). But, if you set the root logger’s level to DEBUG, indeed the message get sent to the log file:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, image that you want to add a new handler that doesn’t record debug information.
You can do this by simply setting the handler logging level:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
h2 = logging.handlers.RotatingFileHandler('foo2.log')
h2.setLevel(logging.WARNING)
a.addHandler(h2)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, the log file foo.log will contain both messages, while the file foo2.log will only contain the warning message. You could be interested in having a log file of only error-level messages, then simply add a Handler and set its level to logging.ERROR, everything using the same Logger.
You may think of the Logger logging level as a global restriction on which messages are “interesting” for a given logger and its handlers. The messages that are considered by the logger afterwards get sent to the handlers, which perform their own filtering and logging process.
In Python logging there are two different concepts: the level that the logger logs at and the level that the handler actually activates.
When a call to log is made, what is basically happening is:
if self.level <= loglevel:
for handler in self.handlers:
handler(loglevel, message)
While each of those handlers will then call:
if self.level <= loglevel:
# do something spiffy with the log!
If you’d like a real-world demonstration of this, you can look at Django’s config settings. I’ll include the relevant code here.
LOGGING = {
#snip
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'logging.NullHandler',
},
'console':{
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'filters': ['special']
}
},
'loggers': {
#snip
'myproject.custom': {
# notice how there are two handlers here!
'handlers': ['console', 'mail_admins'],
'level': 'INFO',
'filters': ['special']
}
}
}
So, in the configuration above, only logs to getLogger('myproject.custom').info and above will get processed for logging. When that happens, the console will output all of the results (it will output everything because it is set to DEBUG level), while the mail_admins logger will happen for all ERRORs, FATALs and CRITICALs.
I suppose some code which isn’t Django might help too:
import logging.handlers as hand
import logging as logging
# to make things easier, we'll name all of the logs by the levels
fatal = logging.getLogger('fatal')
warning = logging.getLogger('warning')
info = logging.getLogger('info')
fatal.setLevel(logging.FATAL)
warning.setLevel(logging.WARNING)
info.setLevel(logging.INFO)
fileHandler = hand.RotatingFileHandler('rotating.log')
# notice all three are re-using the same handler.
fatal.addHandler(fileHandler)
warning.addHandler(fileHandler)
info.addHandler(fileHandler)
# the handler should log everything except logging.NOTSET
fileHandler.setLevel(logging.DEBUG)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Debug " + logger.name + " = " + level)
# now, the handler will only do anything for *fatal* messages...
fileHandler.setLevel(logging.FATAL)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Fatal " + logger.name + " = " + level)
That results in:
Debug fatal = fatal
Debug warning = warning
Debug warning = error
Debug warning = fatal
Debug info = info
Debug info = warning
Debug info = error
Debug info = fatal
Fatal fatal = fatal
Fatal warning = fatal
Fatal info = fatal
Again, notice how info logged something at info, warning, error, and fatal when the log handler was set to DEBUG, but when the handler was set to FATAL all of a sudden only FATAL messages made it to the file.
Handlers represent different audiences for logging events. Levels on handlers are used to control the verbosity of output seen by a particular audience, and act in addition to any levels set on loggers. Levels on loggers are used to control the overall verbosity of logging from different parts of an application or library.
See this diagram for more information about how logging events are handled:
the rule
if and only if
handler.level <= message.level
&&
logger.level <= message.level
then the message prints.
Reminder: lower values are more verbose
Level | Numeric value
---------|--------------
CRITICAL | 50
ERROR | 40
WARNING | 30
INFO | 20
DEBUG | 10
NOTSET | 0
ref: https://docs.python.org/3/library/logging.html#logging-levels
in other words
if the logger is set to WARNING, it won’t matter if the handler has a more verbose setting. it’ll already be filtered by the time it gets to the handler.
a full example
import logging
handler_info = logging.StreamHandler()
handler_info.setLevel("INFO")
handler_info.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_info: %(message)s"))
handler_debug = logging.StreamHandler()
handler_debug.setLevel("DEBUG")
handler_debug.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_debug: %(message)s"))
logger_info = logging.getLogger('logger_info')
logger_info.setLevel("INFO")
logger_info.addHandler(handler_info)
logger_info.addHandler(handler_debug)
logger_debug = logging.getLogger('logger_debug')
logger_debug.setLevel("DEBUG")
logger_debug.addHandler(handler_info)
logger_debug.addHandler(handler_debug)
print()
print("output for `logger_info.info('hello')`")
logger_info.info("hello")
print()
print("output for `logger_info.debug('bonjour')`")
logger_info.debug("bonjour")
print()
print("output for `logger_debug.info('hola')`")
logger_debug.info("hola")
print()
print("output for `logger_debug.debug('ciao')`")
logger_debug.debug("ciao")
print()
which gives
output for `logger_info.info('hello')`
INFO message for logger_info handled by handler_info: hello
INFO message for logger_info handled by handler_debug: hello
output for `logger_info.debug('bonjour')`
# nothing, because message.level < logger.level
output for `logger_debug.info('hola')`
INFO message for logger_debug handled by handler_info: hola
INFO message for logger_debug handled by handler_debug: hola
output for `logger_debug.debug('ciao')`
DEBUG message for logger_debug handled by handler_debug: ciao
# nothing from handler_info, because message.level < handler.level
Let’s say I have the following code:
import logging
import logging.handlers
a = logging.getLogger('myapp')
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
# The effective log level is still logging.WARN
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
I expect that setting logging.DEBUG on the handler would cause debug-level messages to be written to the log file. However, this prints 30 for the effective level (equal to logging.WARNING, the default), and only logs the warn message to the log file, not the debug message.
It appears that the handler’s log level is being dropped on the floor, e.g. it’s silently ignored. Which makes me wonder, why have setLevel on the handler at all?
It allows finer control. By default the root logger has WARNING level set, this means that it wont print messages with lower level(no matter how the handlers’ levels are set!). But, if you set the root logger’s level to DEBUG, indeed the message get sent to the log file:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, image that you want to add a new handler that doesn’t record debug information.
You can do this by simply setting the handler logging level:
import logging
import logging.handlers
a = logging.getLogger('myapp')
a.setLevel(logging.DEBUG) # set root's level
h = logging.handlers.RotatingFileHandler('foo.log')
h.setLevel(logging.DEBUG)
a.addHandler(h)
h2 = logging.handlers.RotatingFileHandler('foo2.log')
h2.setLevel(logging.WARNING)
a.addHandler(h2)
print a.getEffectiveLevel()
a.debug('foo message')
a.warn('warning message')
Now, the log file foo.log will contain both messages, while the file foo2.log will only contain the warning message. You could be interested in having a log file of only error-level messages, then simply add a Handler and set its level to logging.ERROR, everything using the same Logger.
You may think of the Logger logging level as a global restriction on which messages are “interesting” for a given logger and its handlers. The messages that are considered by the logger afterwards get sent to the handlers, which perform their own filtering and logging process.
In Python logging there are two different concepts: the level that the logger logs at and the level that the handler actually activates.
When a call to log is made, what is basically happening is:
if self.level <= loglevel:
for handler in self.handlers:
handler(loglevel, message)
While each of those handlers will then call:
if self.level <= loglevel:
# do something spiffy with the log!
If you’d like a real-world demonstration of this, you can look at Django’s config settings. I’ll include the relevant code here.
LOGGING = {
#snip
'handlers': {
'null': {
'level': 'DEBUG',
'class': 'logging.NullHandler',
},
'console':{
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
'filters': ['special']
}
},
'loggers': {
#snip
'myproject.custom': {
# notice how there are two handlers here!
'handlers': ['console', 'mail_admins'],
'level': 'INFO',
'filters': ['special']
}
}
}
So, in the configuration above, only logs to getLogger('myproject.custom').info and above will get processed for logging. When that happens, the console will output all of the results (it will output everything because it is set to DEBUG level), while the mail_admins logger will happen for all ERRORs, FATALs and CRITICALs.
I suppose some code which isn’t Django might help too:
import logging.handlers as hand
import logging as logging
# to make things easier, we'll name all of the logs by the levels
fatal = logging.getLogger('fatal')
warning = logging.getLogger('warning')
info = logging.getLogger('info')
fatal.setLevel(logging.FATAL)
warning.setLevel(logging.WARNING)
info.setLevel(logging.INFO)
fileHandler = hand.RotatingFileHandler('rotating.log')
# notice all three are re-using the same handler.
fatal.addHandler(fileHandler)
warning.addHandler(fileHandler)
info.addHandler(fileHandler)
# the handler should log everything except logging.NOTSET
fileHandler.setLevel(logging.DEBUG)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Debug " + logger.name + " = " + level)
# now, the handler will only do anything for *fatal* messages...
fileHandler.setLevel(logging.FATAL)
for logger in [fatal,warning,info]:
for level in ['debug','info','warning','error','fatal']:
method = getattr(logger,level)
method("Fatal " + logger.name + " = " + level)
That results in:
Debug fatal = fatal
Debug warning = warning
Debug warning = error
Debug warning = fatal
Debug info = info
Debug info = warning
Debug info = error
Debug info = fatal
Fatal fatal = fatal
Fatal warning = fatal
Fatal info = fatal
Again, notice how info logged something at info, warning, error, and fatal when the log handler was set to DEBUG, but when the handler was set to FATAL all of a sudden only FATAL messages made it to the file.
Handlers represent different audiences for logging events. Levels on handlers are used to control the verbosity of output seen by a particular audience, and act in addition to any levels set on loggers. Levels on loggers are used to control the overall verbosity of logging from different parts of an application or library.
See this diagram for more information about how logging events are handled:
the rule
if and only if
handler.level <= message.level
&&
logger.level <= message.level
then the message prints.
Reminder: lower values are more verbose
Level | Numeric value
---------|--------------
CRITICAL | 50
ERROR | 40
WARNING | 30
INFO | 20
DEBUG | 10
NOTSET | 0
ref: https://docs.python.org/3/library/logging.html#logging-levels
in other words
if the logger is set to WARNING, it won’t matter if the handler has a more verbose setting. it’ll already be filtered by the time it gets to the handler.
a full example
import logging
handler_info = logging.StreamHandler()
handler_info.setLevel("INFO")
handler_info.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_info: %(message)s"))
handler_debug = logging.StreamHandler()
handler_debug.setLevel("DEBUG")
handler_debug.setFormatter(logging.Formatter(
f"%(levelname)s message for %(name)s handled by handler_debug: %(message)s"))
logger_info = logging.getLogger('logger_info')
logger_info.setLevel("INFO")
logger_info.addHandler(handler_info)
logger_info.addHandler(handler_debug)
logger_debug = logging.getLogger('logger_debug')
logger_debug.setLevel("DEBUG")
logger_debug.addHandler(handler_info)
logger_debug.addHandler(handler_debug)
print()
print("output for `logger_info.info('hello')`")
logger_info.info("hello")
print()
print("output for `logger_info.debug('bonjour')`")
logger_info.debug("bonjour")
print()
print("output for `logger_debug.info('hola')`")
logger_debug.info("hola")
print()
print("output for `logger_debug.debug('ciao')`")
logger_debug.debug("ciao")
print()
which gives
output for `logger_info.info('hello')`
INFO message for logger_info handled by handler_info: hello
INFO message for logger_info handled by handler_debug: hello
output for `logger_info.debug('bonjour')`
# nothing, because message.level < logger.level
output for `logger_debug.info('hola')`
INFO message for logger_debug handled by handler_info: hola
INFO message for logger_debug handled by handler_debug: hola
output for `logger_debug.debug('ciao')`
DEBUG message for logger_debug handled by handler_debug: ciao
# nothing from handler_info, because message.level < handler.level