Queries with python logging in Multiprocessing and QueueHandler

Question

Background

I am trying to learn more about multiprocessing in Python. One requirement for my application is sending logs from all processes to a single file. I was going through this tutorial and the Logging cookbook example.

The code consists of a main process that launches another process to listen for logs. To send logs from all child processes and the main process I am using multiprocessing.Queue.

Code

Main process initializing logging queue and launching listener process :

def main():
    
    log_queue = multiprocessing.Queue() 
    logger = logging.getLogger('my_app')
    logger.addHandler(logging.handlers.QueueHandler(log_queue))
    logger.setLevel(logging.DEBUG)

 
    listener = multiprocessing.Process(target=logger_process, args=(log_queue,), daemon=True)
    listener.start()

    # First log
    logger.info(f'Number of cpus {multiprocessing.cpu_count()}')

    time.sleep(2)
    print(f'Queue size is {log_queue.qsize()}')
    ....

if __name__ == '__main__':
    main()

Logger process to fetch from queue:

def logger_process(queue):
    
    logger = logging.getLogger('my_app')
    f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
    h = logging.FileHandler(filename='ptest.log', mode='a')
    h.setFormatter(f)
    logger.addHandler(h)
    logger.setLevel(logging.DEBUG) # log all messages debug and up

    print('started logger process')
   
    while True:
        print(f'logger before: Queue size is {queue.qsize()}')
        msg = queue.get()
        print(f'logger after: Queue size is {queue.qsize()}')
        if msg is None:  # We send this as a sentinel to tell the listener to quit.
            pass #break # Shutdown
            traceback.print_stack()
        else:
            print(f'Log message is {msg}')
            logger.handle(msg)  # No level or filter logic applied - just do it!
            msg = None

Observation

Prints showing the same log line being repeated and queue size repeatedly going 0 -> 1-> 0 -> 1 …

Queue size is 0
started logger process
logger before: Queue size is 1
logger after: Queue size is 0
Log message is <LogRecord: my_app, 20, ./multiproc.py, 235, "Number of cpus 16">
logger before: Queue size is 1
logger after: Queue size is 0
Log message is <LogRecord: my_app, 20, ./multiproc.py, 235, "Number of cpus 16">
logger before: Queue size is 1
logger after: Queue size is 0
Log message is <LogRecord: my_app, 20, ./multiproc.py, 235, "Number of cpus 16">

EDIT:
(Apologies for the step-by-step sleuthing. I am hoping it may be helpful to some other python newbie like me).

I suspected it to have something with the start method being fork() creating a copy of the parent process (I’m running code on Ubuntu) and thus duplicating some contexts, but changing it to spawn() didn’t help.

Then I played around with commenting out lines from the logger process to (hopefully) narrow down on the problem. I found on commenting out logger.handle(), the program was stopped looping.

Some more search on SO (1, 2, this SO answer to inspect the logger) made me reconsider the logger initialization. Here are the findings:

def logger_process(queue):
    # Configure logging infra
    #logger = logging.getLogger('my_app') # <------ causing a loop
    logger = logging.getLogger()          # <------ works normally
    print(vars(logger))
    listloggers()

Output when logger_process is using getLogger() i.e. root :

{'filters': [], 'name': 'root', 'level': 30, 'parent': None, 'propagate': True, 'handlers': [], 'disabled': False, '_cache': {}}
<RootLogger root (DEBUG)>
     <FileHandler /home/ptest.log (NOTSET)>
+ [my_app              ] <Logger my_app (DEBUG)>
     <QueueHandler (NOTSET)>

Output when logger_process is using getLogger(‘myapp’) :

{'filters': [], 'name': 'my_app', 'level': 10, 'parent': <RootLogger root (WARNING)>, 'propagate': True, 'handlers': [<QueueHandler (NOTSET)>], 'disabled': False, '_cache': {}, 'manager': <logging.Manager object at 0x7f89aa763ee0>}
<RootLogger root (WARNING)>
+ [my_app              ] <Logger my_app (DEBUG)>
     <QueueHandler (NOTSET)>
     <FileHandler /home/ptest.log (NOTSET)>

Questions :

While I can see that propagate is at play here in somehow causing the loop, can someone explain what is exactly happening here?
If I were to create more child processes, what should be their respective getLogger calls? What is the best practice regarding creating loggers – to be or not to be named?
The logging cookbook example shows worker_configurer adding a queue handler to the root logger. Is this because the multiprocessing queue supports multiple producer/consumers and each ‘end’ of the queue needs to be registered in the logging handler?
In the same example above, worker_process() uses this logging.getLogger(choice(LOGGERS)) for each log. Does this mean a new logger is initialized everytime a log is sent? Also, there are 10 workers but 2 LOGGERS : does this mean the same logger will be in different processes? How does that work out in the logger / propagate heirarchy?

Asked By: Panda142308

||

Source

Answer 1

Before it starts the listener process, the master configures a QueueHandler:

logger.addHandler(logging.handlers.QueueHandler(log_queue))

This handler sends all log records to log_queue. Assuming you’re using the fork startmethod (the default on most flavors of Unix), the listener process inherits this setup.

Then your listener process reads records from the queue:

msg = queue.get()

and tells its copy of the logger to handle the record:

logger.handle(msg)

and the worker’s copy of the QueueHandler then dumps the record back into the queue.

If you examine the examples from the logging cookbook, you’ll see that the examples never perform any logging configuration in the master process before starting workers. This ensures workers don’t inherit handlers they’re not supposed to have.

Answered By: user2357112

Queries with python logging in Multiprocessing and QueueHandler

Question:

Answers: