Django – Loggers are overwriting the previous log file along with the new one

Question

I am using Django logger for logging along with gunicorn workers. Currently, I am testing with 3 workers and I am using the TimedRotatingFileHandler. I configured when = m and interval = 1, I tried sending some 1000 requests and it is writing in the new file and overwriting the old file as well.

Here is my config from settings.py

'debug': {
            'level': 'DEBUG',
            'filename': BASE_DIR + '/Log/debug.log',
            'class': 'logging.handlers.TimedRotatingFileHandler',
            'when': 'M',
            'interval': 1,
            'formatter': 'verbose'
        },

This will happen only if i use more than 1 gunicorn worker. But if I use 1 gunicorn worker this is not happening.

Asked By: Suresh

||

Source

Answer 1

You shouldn’t be using file-based handlers directly from multiple processes concurrently, which is what you get if you use gunicorn with this configuration. Instead, use a SocketHandler or QueueHandler to send logs from the webapp to a separate process which logs them to disk. The Python logging cookbook has more information and example code, which is too long to reproduce here.

Writing to a single file from multiple concurrent processes without them treading on each other isn’t supported in Python (and a lot of other languages too).

Answered By: Vinay Sajip

Answer 2

I had the same problem. You could use ConcurrentLogHandler package to fix it.

first install setuptools<58.0.0:

pip install setuptools~=57.5.0

then:

pip install ConcurrentLogHandler

after that change the config like that:

settings.py

from cloghandler import ConcurrentRotatingFileHandler

 'debug': {
        'level': 'DEBUG',
        'filename': BASE_DIR + '/Log/debug.log',
        'class': 'logging.handlers.ConcurrentRotatingFileHandler',
        'formatter': 'verbose'
    },

note that you should remove when and interval.

Answered By: Mohammad Amin Eskandari

Django – Loggers are overwriting the previous log file along with the new one

Question:

Answers: