Python's psycopg2 & logging: Logged at the DEBUG-level queries are recorded in bytes

Question

I have connection to PostgreSQL database:

LOGGER = Logger().LOGGER
# ...
self.connection = psycopg2.connect(connection_factory=LoggingConnection,
                                   dbname=...,
                                   user=...,
                                   password=...,
                                   host=...,
                                   options="-c search_path=...")
self.connection.initialize(LOGGER)
self.cursor = self.connection.cursor()

It’s Logger class:

# some data

_MODE = "w"
_ENCODING = "utf-8"
_FORMATTER = "%(asctime)s %(levelname)s [%(filename)s:%(lineno)s] %(message)s"

class Logger:
    @property
    def LOGGER(self):
        logger = logging.getLogger()
        logger.setLevel(logging.DEBUG)

        fileHandler = logging.FileHandler(filename=_FILE,
                                          mode=_MODE,
                                          encoding=_ENCODING)
        fileHandler.setFormatter(logging.Formatter(_FORMATTER))
        logger.addHandler(fileHandler)

        return logger
# ...

When I execute query, it’s recorded to LOG file in bytes, not casual string, so cyrillic characters are not displayed exclicitly. Here is an example:

2022-12-15 03:47:59,914 DEBUG [extras.py:427] b"INSERT INTO test VALUES (1, 'xd0x9axd0xb8xd1x80xd0xb8xd0xbbxd0xbbxd0xb8xd1x86xd0xb0')"

which corresponds to: INSERT INTO test VALUES (1, 'Кириллица') (just some test data).

So is there any way to decode bytes? Everything is fine with logging another events not related to executing queries (DEBUG).

Thanks in advance.

Expecting something like this:

2022-12-15 03:47:59,914 DEBUG [extras.py:427] INSERT INTO test VALUES (1, 'Кириллица')

Asked By: Emma Grinberg

||

Source

Answer 1

P.S. I decided not to use LoggingConnection and created method which execute queries and log them the way I want:

def execute(self, query: str) -> tuple | None:
    """
    """
    from contextlib import closing
    with closing(self.connection) as connection:
        with connection.cursor() as cursor:
            try:
                cursor.execute(query)
                LOGGER.debug(query)
            except:
                # catch my exceptions here

I don’t know how competent this solutions is, but it works perfectly for me.

Answered By: Emma Grinberg

Answer 2

You can add a filter to the logger to decode the bytes:

class DecodingFilter:
    # Provide the correct encoding if your connection is not using UTF-8.
    def __init__(self, encoding='utf-8'):
        self.encoding = encoding

    def filter(self, record):
        record.msg = record.msg.decode(self.encoding)
        return record

logger.addFilter(DecodingFilter())

For me, using Python 3.11 and psycopg2 2.9.5, this test code:

logger = logging.getLogger('test_logger')
logger.addFilter(DecodingFilter())
logger.setLevel(logging.DEBUG)
logging.basicConfig()

with psycopg2.connect(dbname='test', connection_factory=extras.LoggingConnection) as conn:
    conn.initialize(logger)
    cur = conn.cursor()
    cur.execute('SELECT 1')

generates this output:

DEBUG:test_logger:SELECT 1

Answered By: snakecharmerb

Python's psycopg2 & logging: Logged at the DEBUG-level queries are recorded in bytes

Question:

Answers: