Multiline log records in syslog

Question

So I’ve configured my Python application to log to syslog with Python’s SysLogHandler, and everything works fine. Except for multi-line handling. Not that I need to emit multiline log records so badly (I do a little), but I need to be able to read Python’s exceptions. I’m using Ubuntu with rsyslog 4.2.0. This is what I’m getting:

Mar 28 20:11:59 telemachos root: ERROR 'EXCEPTION'#012Traceback (most recent call last):#012  File "./test.py", line 22, in <module>#012    foo()#012  File "./test.py", line 13, in foo#012    bar()#012  File "./test.py", line 16, in bar#012    bla()#012  File "./test.py", line 19, in bla#012    raise Exception("EXCEPTION!")#012Exception: EXCEPTION!

Test code in case you need it:

import logging
from logging.handlers import SysLogHandler

logger = logging.getLogger()
logger.setLevel(logging.INFO)
syslog = SysLogHandler(address='/dev/log', facility='local0')
formatter = logging.Formatter('%(name)s: %(levelname)s %(message)r')
syslog.setFormatter(formatter)
logger.addHandler(syslog)

def foo():
    bar()

def bar():
    bla()

def bla():
    raise Exception("EXCEPTION!")

try:
    foo()
except:
    logger.exception("EXCEPTION")

Asked By: Shay Rojansky

||

Source

Answer 1

OK, figured it out finally…

rsyslog by default escapes all weird characters (ASCII < 32), and this include newlines (as well as tabs and others).

$EscapeControlCharactersOnReceive:

This directive instructs rsyslogd to replace control characters during reception of the
message. The intent is to provide a way to stop non-printable
messages from entering the syslog system as whole. If this option is
turned on, all control-characters are converted to a 3-digit octal
number and be prefixed with the $ControlCharacterEscapePrefix
character (being ‘’ by default). For example, if the BEL character
(ctrl-g) is included in the message, it would be converted to “07”.

You can simply add this to your rsyslog config to turn it off:

$EscapeControlCharactersOnReceive off

or, with the "new" advanced syntax:

global(parser.escapeControlCharactersOnReceive="off")

Answered By: Shay Rojansky

Answer 2

Alternatively, if you want to keep your syslog intact on one line for parsing, you can just replace the characters when viewing the log.

tail -f /var/log/syslog | sed 's/#012/nt/g'

Answered By: Nick Zalutskiy

Answer 3

Another option would be to subclass the SysLogHandler and override emit() – you could then call the superclass emit() for each line in the text you’re sent. Something like:

from logging import LogRecord
from logging.handlers import SysLogHandler

class MultilineSysLogHandler(SysLogHandler):
    def emit(self, record):
        if 'n' in record.msg:
            record_args = [record.args] if isinstance(record.args, dict) else record.args
            for single_line in record.msg.split('n'):
                single_line_record = LogRecord(
                    name=record.name,
                    level=record.levelno,
                    pathname=record.pathname,
                    msg=single_line,
                    args=record_args,
                    exc_info=record.exc_info,
                    func=record.funcName
                )
                super(MultilineSysLogHandler, self).emit(single_line_record)
        else:
            super(MultilineSysLogHandler, self).emit(record)

Answered By: Hamish Downer

Multiline log records in syslog

Question:

Answers: