Logging UUID per API request in Python FastAPI

Question

I have a pure python package(let’s call it main) that has a few functions for managing infrastructure. Alongside, I have created a FastAPI service that can make calls to the main module to invoke functionality as per need.

For logging, I’m using loguru. The API on startup creates a loguru instance, settings are applied and a generic UUID is set (namely, [main]). On every incoming request to the API, a pre_request function generates a new UUID and calls the loguru to configure with that UUID. At the end of the request, the UUID is set back to default UUID [main].

The problem that I’m facing is on concurrent requests, the new UUID takes over and all the logs are now being written with the UUID that was configured latest. Is there a way I can instantiate the loguru module on every request and make sure there’s no cross logging happening for parallelly processed API requests?

Implementation:

In init.py of the main package:

from loguru import logger 
logger.remove() #to delete all existing default loggers 
logger.add(filename, format, level, retention, rotation)  #format
logger.configure(extra={"uuid": "main"})

In all modules, the logger is imported as

from loguru import logger

In api/ package – on every new request, I have this below code block:

uuid = get_uuid() #calling util func to get a new uuid
logger.configure(uuid=uuid) 
# Here onwards, all log messages contain this uuid 
# At the end of the request, I configure it back to default uuid (i.e. "main")

The configure method is updating the root logger, I tried using the bind method instead, which according to the loguru docs, can be used to contextualize extra record attributes, but it does not seem to have any effect (I still see the default UUID, i.e. "main", only when I use .configure the UUID gets set).

Any ideas on how should I go about setting the UUID, so that all concurrent requests to the API have their own UUID? Since there are multiple sub-modules that get called to serve one API request and all of them have some logging in it, I need the UUID to persist for all the modules per request. It seems like I need to have a logger instance per API request, but I am not sure how to instantiate it correctly to make this work.

The current implementation works if the API is serving one request, but the logging breaks when serving more than 1 call (since the UUID that gets logged is the last one that as configured)

Asked By: aafre

||

Source

Answer 1

I created this middleware, which before routing calls, configures the logger instance with the UUID with the user of context manager:

from loguru import logger
from contextvars import ContextVar
from starlette.middleware.base import BaseHTTPMiddleware

_request_id = ContextVar("request_id", default=None)


def get_request_id():
    return _request_id.get()

class ContextualizeRequest(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        uuid = get_uuid()
        request_id = _request_id.set(uuid)  # set uuid to context variable
        with logger.contextualize(uuid=get_request_id()):
            try:
                response = await call_next(request)
            except Exception:
                logger.error("Request failed")
            finally:
                _request_id.reset()
                return response

The only caveat is, if you use multithreading, multi-processing, or create new event loops (in internal calls), the logger instance in that block will not have this UUID.

Answered By: aafre

Answer 2

Have a file that contains a contextvar (that you import and set from whatever file you want).
In the same file, add a method that adds this contextvar to LogRecord Objects by overriding logging.getLogRecordFactory.

The advantage of this solution is that you can use logging.basicConfig normally. No loggers need to be configured, and is simple.

Sample implementatiom

Answered By: arrmansa

Logging UUID per API request in Python FastAPI

Question:

Answers: