Python logging – Is there something below DEBUG?
Question:
In some other technologies we occasionally used a log level below DEBUG that I believe was called “verbose”. I realize that the need for such a level is very subjective. But in my mind “just” having INFO and DEBUG isn’t really enough. We had times where something very spammy (more spammy than debug) needed to be logged. In practice we’d produce builds without this turned on but in a few occasions we’d enable this level of logging after the product was all installed on some QA setup, while tracking down a bug, etc.
Is there any way (easy or otherwise) to log something below the DEBUG level using the standard python logging library?
In a temp.py file I can do the following:
logging.addLevelName(5,"verbose")
VERBOSE = 5
logger = logging.getLogger("foo")
logger.setLevel(VERBOSE)
logger.log(VERBOSE,"blah!")
This works when I run temp.py within my IDE (and logs to stdout) but our real daemons use the standard file/dictionary configuration syntax to setup logging and I don’t see any way to indicate that level 5 should be used for the daemon.
Am I chasing something that’s not really feasible?
For those who might wonder why I’d need anything lower than DEBUG, it’s for the occasional type of logging that might occur very frequently (maybe an inner loop) that I wouldn’t normally want to see even at DEBUG but on some production system it might be helpful to enable it once in awhile without needing to add more logging to the source code and re-deploy, etc.
EDIT1 – Clearly the logging library allows for custom levels. Since DEBUG is level 10, there’s room somewhere in the 1..9 range. If I define a custom level (such as in the sample code above), I guess my real question is how do I enable that level of logging from the json log configuration file?
EDIT2 – The following would work if it weren’t for the fact that we need/use json configuration files (a requirement that I cannot change):
import logging
logging.basicConfig(filename='example.log',level=5)
VERBOSE = 5
logging.addLevelName(5,"verbose")
logger = logging.getLogger("bar")
logger.log(VERBOSE,"blah!")
EDIT3 – Figured it out… The call to
logging.addLevelName(5,"VERBOSE")
is critical. I just didn’t have it in the right place. In my case I just needed to make the above call takes place before the call to the logging libraries dictConfig(…) call. After I did this I was then able to go into our log configuration file and bump things down (on both the file handler and the root) to VERBOSE and it worked.
Granted, the log statement itself isn’t exactly elegant because you call:
self.logger.log(VERBOSE,"Something very spammy")
rather than
self.logger.verbose("Something very spammy")
But I really didn’t want to modify any logger library code (been there, done that, have the t-shirt).
Thanks all!
And to those who think nothing lower than DEBUG is needed, more power to you 🙂
Answers:
DEBUG is the lowest level out of the ones provided by the logging module: ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'). Their numeric values are here: http://docs.python.org/howto/logging.html#logging-levels
You can create custom levels (though the docs say that that should rarely be necessary and may even be undesirable). If you want to add a level, the technique is simple:
>>> logging.addLevelName(5, "VERBOSE")
Eventhough you can add a custom level, it may be a better approach to add some filters that provide a finer level of control.
You can even go further and add a logger.verbose method, although I strongly suggest you don’t for various reasons (pretty much covered in logging’s how-to). Anyway, if you decide that you really want to have one, here’s the code:
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
The answer from @voitek works great, but he forgot to monkey patch logging.verbose.
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.verbose = lambda msg, *args, **kwargs: logging.log(logging.VERBOSE, msg, *args, **kwargs)
This will now also work with;
logging.verbose(*args, **kwargs)
Adding to @sleepycal’s answer, you might also want to add a verbose method to the LoggerAdapter:
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.LoggerAdapter.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.verbose = lambda msg, *args, **kwargs: logging.log(logging.VERBOSE, msg, *args, **kwargs)
In some other technologies we occasionally used a log level below DEBUG that I believe was called “verbose”. I realize that the need for such a level is very subjective. But in my mind “just” having INFO and DEBUG isn’t really enough. We had times where something very spammy (more spammy than debug) needed to be logged. In practice we’d produce builds without this turned on but in a few occasions we’d enable this level of logging after the product was all installed on some QA setup, while tracking down a bug, etc.
Is there any way (easy or otherwise) to log something below the DEBUG level using the standard python logging library?
In a temp.py file I can do the following:
logging.addLevelName(5,"verbose")
VERBOSE = 5
logger = logging.getLogger("foo")
logger.setLevel(VERBOSE)
logger.log(VERBOSE,"blah!")
This works when I run temp.py within my IDE (and logs to stdout) but our real daemons use the standard file/dictionary configuration syntax to setup logging and I don’t see any way to indicate that level 5 should be used for the daemon.
Am I chasing something that’s not really feasible?
For those who might wonder why I’d need anything lower than DEBUG, it’s for the occasional type of logging that might occur very frequently (maybe an inner loop) that I wouldn’t normally want to see even at DEBUG but on some production system it might be helpful to enable it once in awhile without needing to add more logging to the source code and re-deploy, etc.
EDIT1 – Clearly the logging library allows for custom levels. Since DEBUG is level 10, there’s room somewhere in the 1..9 range. If I define a custom level (such as in the sample code above), I guess my real question is how do I enable that level of logging from the json log configuration file?
EDIT2 – The following would work if it weren’t for the fact that we need/use json configuration files (a requirement that I cannot change):
import logging
logging.basicConfig(filename='example.log',level=5)
VERBOSE = 5
logging.addLevelName(5,"verbose")
logger = logging.getLogger("bar")
logger.log(VERBOSE,"blah!")
EDIT3 – Figured it out… The call to
logging.addLevelName(5,"VERBOSE")
is critical. I just didn’t have it in the right place. In my case I just needed to make the above call takes place before the call to the logging libraries dictConfig(…) call. After I did this I was then able to go into our log configuration file and bump things down (on both the file handler and the root) to VERBOSE and it worked.
Granted, the log statement itself isn’t exactly elegant because you call:
self.logger.log(VERBOSE,"Something very spammy")
rather than
self.logger.verbose("Something very spammy")
But I really didn’t want to modify any logger library code (been there, done that, have the t-shirt).
Thanks all!
And to those who think nothing lower than DEBUG is needed, more power to you 🙂
DEBUG is the lowest level out of the ones provided by the logging module: ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'). Their numeric values are here: http://docs.python.org/howto/logging.html#logging-levels
You can create custom levels (though the docs say that that should rarely be necessary and may even be undesirable). If you want to add a level, the technique is simple:
>>> logging.addLevelName(5, "VERBOSE")
Eventhough you can add a custom level, it may be a better approach to add some filters that provide a finer level of control.
You can even go further and add a logger.verbose method, although I strongly suggest you don’t for various reasons (pretty much covered in logging’s how-to). Anyway, if you decide that you really want to have one, here’s the code:
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
The answer from @voitek works great, but he forgot to monkey patch logging.verbose.
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.verbose = lambda msg, *args, **kwargs: logging.log(logging.VERBOSE, msg, *args, **kwargs)
This will now also work with;
logging.verbose(*args, **kwargs)
Adding to @sleepycal’s answer, you might also want to add a verbose method to the LoggerAdapter:
logging.VERBOSE = 5
logging.addLevelName(logging.VERBOSE, "VERBOSE")
logging.Logger.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.LoggerAdapter.verbose = lambda inst, msg, *args, **kwargs: inst.log(logging.VERBOSE, msg, *args, **kwargs)
logging.verbose = lambda msg, *args, **kwargs: logging.log(logging.VERBOSE, msg, *args, **kwargs)