Simple Python Logging Exception from Future

Question

This should be a really simple question, but after googling, reading docs, and several other SO threads, I don’t see the answer: How do I log an exception with Python standard logging? One small wrinkle is that I’m getting the exception from a Future. I’m not writing the except exception handler myself. Ideally, I would get the exception message, a stack trace, the extra message sent, and maybe the type of exception. Here’s a simple program that shows my issue:

import logging
from concurrent.futures import ThreadPoolExecutor

logger = logging.getLogger(__name__)


def test_f(a, b=-99, c=50):
    logger.info("test_f a={} b={} c={}".format(a, b, c))


def future_callback_error_logger(future):
    e = future.exception()
    if e is not None:
        # This log statement does not seem to do what I want.
        # It logs "Executor Exception" with no information about the exception.
        # I would like to see the exception type, message, and stack trace.
        logger.error("Executor Exception", exc_info=e)


def submit_with_log_on_error(executor, func, *args, **kwargs):
    future = executor.submit(func, *args, **kwargs)
    future.add_done_callback(future_callback_error_logger)


if __name__ == "__main__":
    logging.basicConfig(level="DEBUG")

    logger.info("start")
    executor = ThreadPoolExecutor(max_workers=5)

    # This will work.
    submit_with_log_on_error(executor, test_f, 10, c=20)
    # This will intentionally trigger an error due to too many arguments.
    # I would like that error to be properly logged.
    submit_with_log_on_error(executor, test_f, 10, 20, 30, 40)
    # This will work.
    submit_with_log_on_error(executor, test_f, 50, c=60)

    executor.shutdown(True)
    logger.info("shutdown")

Asked By: clay

||

Source

Answer 1

To use logger.exception and get the traceback etc, you need to be inside an except block. Instead of checking the future.exception(), which returns the exception (if any), use the future.result() which raises the exception (if any).

So, try this instead (if you’ll pardon the pun):

def future_callback_error_logger(future):
    try:
        future.result()
    except Exception:
        logger.exception("Executor Exception")

Answered By: wim

Simple Python Logging Exception from Future

Question:

Answers: