Extract the extra fields in logging call in log formatter

Question

so I can add additional fields to my log message like so

logging.info("My log Message", extra={"someContext":1, "someOtherContext":2})

which is nice

but unclear how to extract all the extra fields in my log formatter

def format(self, record):
  record_dict = record.__dict__.copy()
  print(record_dict)

in the above i can see all my extra fields in the output dict but they are flattened in to a dict with load of other junk i dont want

{'name': 'root', 'msg': 'My log Message', 'args': (), 'levelname': 'INFO', 'levelno': 20, 'pathname': '.\handler.py', 'filename': 'handler.py', 'module': 'handler', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 27, 'funcName': 'getPlan', 'created': 1575461352.0664868, 'msecs': 66.48683547973633, 'relativeCreated': 1253.0038356781006, 'thread': 15096, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 23740, 'someContext': 1, 'someOtherContext':2}

is there any way of getting all my extra keys without having to know them all upfront,

i am writing a json formatter and want to create a dict a la

justMyExtra = ?????
to_log = {
"message" record_dict["message"], 
**justMyExtra
}

Asked By: Mingo

||

Source

Answer 1

Two approaches come to mind:

Dump all of the extra fields into a dictionary within your main dictionary. Call the key “additionalContext” and get all the extra entries.
Create a copy of the original dictionary and delete all of your known keys: ‘name’,’msg’,’args’, etc. until you only have justYourExtra

Answered By: james hendricks

Answer 2

If you read the source code of the logging.Logger.makeRecord method, which returns a LogRecord object with the given logging information, you’ll find that it merges the extra dict with the __dict__ attribute of the returing LogRecord object, so you cannot retrieve the original extra dict afterwards in the formatter.

Instead, you can patch the logging.Logger.makeRecord method with a wrapper function that stores the given extra dict as the _extra attribute of the returning LogRecord object:

def make_record_with_extra(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None):
    record = original_makeRecord(self, name, level, fn, lno, msg, args, exc_info, func, extra, sinfo)
    record._extra = extra
    return record

original_makeRecord = logging.Logger.makeRecord
logging.Logger.makeRecord = make_record_with_extra

so that:

class myFormatter(logging.Formatter):
    def format(self, record):
        print('Got extra:', record._extra) # or do whatever you want with _extra
        return super().format(record)

logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(myFormatter('%(name)s - %(levelname)s - %(message)s - %(foo)s'))
logger.addHandler(handler)
logger.warning('test', extra={'foo': 'bar'})

outputs:

Got extra: {'foo': 'bar'}
__main__ - WARNING - test - bar

Demo: https://repl.it/@blhsing/WorthyTotalLivedistro

Answered By: blhsing

Answer 3

Two approaches similar to those from @james-hendricks, but a little less fragile:

Use the existing args dict on the LogRecord instead of coming up with your own dict.
Create a dummy LogRecord and inspect its keys to get a non-fragile str list of LogRecord special keys:

  dummy = logging.LogRecord('dummy', 0, 'dummy', 0, None, None, None, None, None)
  reserved_keys = dummy.__dict__.keys()

Answered By: John Vandivier

Extract the extra fields in logging call in log formatter

Question:

Answers: