Is Python's logging module thread safe?
Question:
If you call the same logging handler from two different python threads, is there a need for locking?
Answers:
The logging module is thread-safe; it handles the locking for you. See the docs.
If you call the same handler from different threads, it is thread safe. And I’ll just extend a little bit for this question. In fact, it depends on how you use the logging module. There still are some not-thread-safe circumstances when logging in multiple threads.
If multiple threads use different logger instances (e.g., calling the logging.getLogger() with different name in each threads), and those logger instances have their own FileHandler that points to the same file, it will result in a race condition and thus not thread-safe anymore.
Also, if multiple logger instances in different threads hold their own FileRotatingHandler or TimedRotatingFileHandler, while they are pointing to the same file (i.e., giving the same file name when instantiating them), the logics they rotate the file are not thread safe, neither. When they need to rotate the file, something strange could happen. (You can refer to the question Python TimedRotatingFileHandler – logs are missing, which cause by a similar reason)
Under the hood, in the logging module, each handler instance holds an threading.RLock instance, hence different loggers that hold different handlers will hold different RLocks, so those locks cannot avoid the race condition when loggers try to write to the same file in different threads — loggers are still able to write bytes at the same time to the same file even they have acquired their own RLock.
You can refer to the Logging Cookbook – Opening the same log file multiple times part for more information.
If you call the same logging handler from two different python threads, is there a need for locking?
The logging module is thread-safe; it handles the locking for you. See the docs.
If you call the same handler from different threads, it is thread safe. And I’ll just extend a little bit for this question. In fact, it depends on how you use the logging module. There still are some not-thread-safe circumstances when logging in multiple threads.
If multiple threads use different logger instances (e.g., calling the logging.getLogger() with different name in each threads), and those logger instances have their own FileHandler that points to the same file, it will result in a race condition and thus not thread-safe anymore.
Also, if multiple logger instances in different threads hold their own FileRotatingHandler or TimedRotatingFileHandler, while they are pointing to the same file (i.e., giving the same file name when instantiating them), the logics they rotate the file are not thread safe, neither. When they need to rotate the file, something strange could happen. (You can refer to the question Python TimedRotatingFileHandler – logs are missing, which cause by a similar reason)
Under the hood, in the logging module, each handler instance holds an threading.RLock instance, hence different loggers that hold different handlers will hold different RLocks, so those locks cannot avoid the race condition when loggers try to write to the same file in different threads — loggers are still able to write bytes at the same time to the same file even they have acquired their own RLock.
You can refer to the Logging Cookbook – Opening the same log file multiple times part for more information.