What is a correct way to filter different loggers using python logging?
Question:
My purpose is to do a multi-module logging with hierarchical filtering
the way it is proposed by logging author Vinay Sajip, at least as far as I guess 😉
You can skip to “How I want it to work“
Unfortunately, I learned very quickly that working with logging facilities is much more sophisticated than most of my other experience with the language and I already did a lot of common (design) mistakes, e.g. trying to achieve a centralized single Logger class logging for multiple modules or even schemes like (using Python logger class to generate multiple logs for different log levels). But apparently there is room for better design, and it might be worse spending time finding and learning it. So, right now I hope I am on the right track. Otherwise Vinaj will have to clarify the rest 😉
I arrange my logging as this:
- Each python module has its own logger
- Each logger has a name same as the module where it is defined, e.g.
logger = logging.getLogger(__name__)
- Like this, code inside each module can use its own (locally defined) logger to send logging messages (logging.LogRecord) to handlers (logging.Handler)
- Use logging.config to achieve full flexibility in configuring of the logging (Note: in the code below I just start with basicConfig)
Such approach is a recommended approach and I agree with its possible advantages. For example I can turn on/off DEBUG of external libraries using fully qualified module names (the naming hierarchy which already exists in the code).
Now to have a greater level of control I want to use logging.Filter class, to be able to filter (allow) only a selected subtree within the hierarchy of loggers.
This is all fine, but the filtering as described here
Filter instances are used to perform arbitrary filtering of LogRecords.
Loggers and Handlers can optionally use Filter instances to filter
records as desired. The base filter class only allows events which are
below a certain point in the logger hierarchy. For example, a filter
initialized with "A.B" will allow events logged by loggers "A.B",
"A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If
initialized with the empty string, all events are passed.
is still not working for me.
My guess is that my lack of understanding the details behind LogRecords propagation is the source of the problem. Before jumping to code I want to show here a flow chart (from the cookbook tutorial which at first I somehow failed to immediately discover):
Example code
I start with two modules example, each uses it own named logger:
bar.py:
import logging
logger = logging.getLogger(__name__)
def bar():
logger.info('hello from ' + __name__)
foo.py:
import logging
from bar import bar, logger as bar_logger
logger = logging.getLogger('foo')
def foo():
logger.info('hello from foo')
if __name__ == '__main__':
# Trivial logging setup.
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M'
)
# Do some work.
foo()
bar()
Logging is first constructed with logging.basicConfig (root logger, which was created after import logging by __main__ gets a stream handler attached to it, so that we have a console), is enabled (respective Logger.disabled=False) and both module loggers bar and foo propagate to the root logger (so we have three loggers in total).
print logger
print bar_logger
print logging.root
# Prints
#<logging.Logger object at 0x7f0cfd520790>
#<logging.Logger object at 0x7f0cfd55d710>
#<logging.RootLogger object at 0x7f0cfd520550>
The actual usecase is when bar is an external library that I want to silence (filter out).
How it works, but “me” don’t like it
# Don't like it
bar_logger.addFilter(logging.Filter('foo'))
# Do some work.
foo()
bar()
prints only
06-24 14:08 foo INFO hello from foo
How I want it to work
I want to filter it out centrally, i.e. at my root logger w/o the need to import all the loggers of all the external modules out there.
logging.root.addFilter(logging.Filter('foo'))
prints
06-24 14:17 foo INFO hello from foo
06-24 14:17 bar INFO hello from bar
There must be some obvious/stupid mistake that I miss: I don’t want any messages from bar logger. Hey, but what is a better way to find it than summarizing all on SO, folks? 😉
I will try to find out the way for bar_logger to wait for decision from the root logger, before emitting anything. I just hope that this is indeed how it is supposed to work in the first place.
Answers:
Solution
Add the filter to the handler rather than the logger:
handler.addFilter(logging.Filter('foo'))
Explanation
In the flow chart diagram you posted, notice there are two diamonds:
- Does a filter attached to logger reject the record?
- Does a filter attached to hander reject the record?
Thus, you get two swings at rejecting a LogRecord. If you attach the filter to the root logger, but initiate the LogRecord through, say, the foo or bar loggers, then the LogRecord does not get filtered because the LogRecord passes through the foo or bar loggers freely and the root logger filter never enters into play. (Look at the flow chart again.)
In contrast, the StreamHandler defined by basicConfig is capable of filtering any LogRecord passes to it.
So: add the filter to the handler rather than the logger:
# foo.py
import logging
import bar
logger = logging.getLogger('foo')
def foo():
logger.info('hello from foo')
if __name__ == '__main__':
# Trivial logging setup.
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
for handler in logging.root.handlers:
handler.addFilter(logging.Filter('foo'))
foo()
bar.bar()
yields
06-24 09:17 foo INFO hello from foo
If you want to allow logging from loggers whose name begins with foo or bar, but not from any other loggers, you could create a whitelist filter like this:
import logging
foo_logger = logging.getLogger('foo')
bar_logger = logging.getLogger('bar')
baz_logger = logging.getLogger('baz')
class Whitelist(logging.Filter):
def __init__(self, *whitelist):
self.whitelist = [logging.Filter(name) for name in whitelist]
def filter(self, record):
return any(f.filter(record) for f in self.whitelist)
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
for handler in logging.root.handlers:
handler.addFilter(Whitelist('foo', 'bar'))
foo_logger.info('hello from foo')
# 06-24 09:41 foo INFO hello from foo
bar_logger.info('hello from bar')
# 06-24 09:41 bar INFO hello from bar
baz_logger.info('hello from baz')
# No output since Whitelist filters if record.name not begin with 'foo' or 'bar'
And similarly, you could blacklist logger names with this:
class Blacklist(Whitelist):
def filter(self, record):
return not Whitelist.filter(self, record)
My purpose is to do a multi-module logging with hierarchical filtering
the way it is proposed by logging author Vinay Sajip, at least as far as I guess 😉
You can skip to “How I want it to work“
Unfortunately, I learned very quickly that working with logging facilities is much more sophisticated than most of my other experience with the language and I already did a lot of common (design) mistakes, e.g. trying to achieve a centralized single Logger class logging for multiple modules or even schemes like (using Python logger class to generate multiple logs for different log levels). But apparently there is room for better design, and it might be worse spending time finding and learning it. So, right now I hope I am on the right track. Otherwise Vinaj will have to clarify the rest 😉
I arrange my logging as this:
- Each python module has its own logger
- Each logger has a name same as the module where it is defined, e.g.
logger = logging.getLogger(__name__) - Like this, code inside each module can use its own (locally defined) logger to send logging messages (logging.LogRecord) to handlers (logging.Handler)
- Use logging.config to achieve full flexibility in configuring of the logging (Note: in the code below I just start with basicConfig)
Such approach is a recommended approach and I agree with its possible advantages. For example I can turn on/off DEBUG of external libraries using fully qualified module names (the naming hierarchy which already exists in the code).
Now to have a greater level of control I want to use logging.Filter class, to be able to filter (allow) only a selected subtree within the hierarchy of loggers.
This is all fine, but the filtering as described here
Filter instances are used to perform arbitrary filtering of LogRecords. Loggers and Handlers can optionally use Filter instances to filter records as desired. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with "A.B" will allow events logged by loggers "A.B", "A.B.C", "A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with the empty string, all events are passed.
is still not working for me.
My guess is that my lack of understanding the details behind LogRecords propagation is the source of the problem. Before jumping to code I want to show here a flow chart (from the cookbook tutorial which at first I somehow failed to immediately discover):
Example code
I start with two modules example, each uses it own named logger:
bar.py:
import logging
logger = logging.getLogger(__name__)
def bar():
logger.info('hello from ' + __name__)
foo.py:
import logging
from bar import bar, logger as bar_logger
logger = logging.getLogger('foo')
def foo():
logger.info('hello from foo')
if __name__ == '__main__':
# Trivial logging setup.
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M'
)
# Do some work.
foo()
bar()
Logging is first constructed with logging.basicConfig (root logger, which was created after import logging by __main__ gets a stream handler attached to it, so that we have a console), is enabled (respective Logger.disabled=False) and both module loggers bar and foo propagate to the root logger (so we have three loggers in total).
print logger
print bar_logger
print logging.root
# Prints
#<logging.Logger object at 0x7f0cfd520790>
#<logging.Logger object at 0x7f0cfd55d710>
#<logging.RootLogger object at 0x7f0cfd520550>
The actual usecase is when bar is an external library that I want to silence (filter out).
How it works, but “me” don’t like it
# Don't like it
bar_logger.addFilter(logging.Filter('foo'))
# Do some work.
foo()
bar()
prints only
06-24 14:08 foo INFO hello from foo
How I want it to work
I want to filter it out centrally, i.e. at my root logger w/o the need to import all the loggers of all the external modules out there.
logging.root.addFilter(logging.Filter('foo'))
prints
06-24 14:17 foo INFO hello from foo
06-24 14:17 bar INFO hello from bar
There must be some obvious/stupid mistake that I miss: I don’t want any messages from bar logger. Hey, but what is a better way to find it than summarizing all on SO, folks? 😉
I will try to find out the way for bar_logger to wait for decision from the root logger, before emitting anything. I just hope that this is indeed how it is supposed to work in the first place.
Solution
Add the filter to the handler rather than the logger:
handler.addFilter(logging.Filter('foo'))
Explanation
In the flow chart diagram you posted, notice there are two diamonds:
- Does a filter attached to logger reject the record?
- Does a filter attached to hander reject the record?
Thus, you get two swings at rejecting a LogRecord. If you attach the filter to the root logger, but initiate the LogRecord through, say, the foo or bar loggers, then the LogRecord does not get filtered because the LogRecord passes through the foo or bar loggers freely and the root logger filter never enters into play. (Look at the flow chart again.)
In contrast, the StreamHandler defined by basicConfig is capable of filtering any LogRecord passes to it.
So: add the filter to the handler rather than the logger:
# foo.py
import logging
import bar
logger = logging.getLogger('foo')
def foo():
logger.info('hello from foo')
if __name__ == '__main__':
# Trivial logging setup.
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
for handler in logging.root.handlers:
handler.addFilter(logging.Filter('foo'))
foo()
bar.bar()
yields
06-24 09:17 foo INFO hello from foo
If you want to allow logging from loggers whose name begins with foo or bar, but not from any other loggers, you could create a whitelist filter like this:
import logging
foo_logger = logging.getLogger('foo')
bar_logger = logging.getLogger('bar')
baz_logger = logging.getLogger('baz')
class Whitelist(logging.Filter):
def __init__(self, *whitelist):
self.whitelist = [logging.Filter(name) for name in whitelist]
def filter(self, record):
return any(f.filter(record) for f in self.whitelist)
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s %(name)-20s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
for handler in logging.root.handlers:
handler.addFilter(Whitelist('foo', 'bar'))
foo_logger.info('hello from foo')
# 06-24 09:41 foo INFO hello from foo
bar_logger.info('hello from bar')
# 06-24 09:41 bar INFO hello from bar
baz_logger.info('hello from baz')
# No output since Whitelist filters if record.name not begin with 'foo' or 'bar'
And similarly, you could blacklist logger names with this:
class Blacklist(Whitelist):
def filter(self, record):
return not Whitelist.filter(self, record)