Multiprocessing debug techniques
Question:
I’m having trouble debugging a multi-process application (specifically using a process pool in python’s multiprocessing module). I have an apparent deadlock and I do not know what is causing it. The stack trace is not sufficient to describe the issue, as it only displays code in the multiprocessing module.
Are there any python tools, or otherwise general techniques used to debug deadlocks?
Answers:
Yah, debugging deadlocks is fun. You can set the logging level to be higher — see the Python documentation for a description of it, but really quickly:
import multiprocessing, logging
logger = multiprocessing.log_to_stderr()
logger.setLevel(multiprocessing.SUBDEBUG)
Also, add logging for anything in your code that deals with a resource or whatnot that might be in contention. Finally, shot in the dark: spawning off child processes during an import might cause a problem.
In order to avoid deadlocks in the first place, learning good practices is useful, as parallel processing is indeed quite subtle. The (free) Little Book of Semaphores can be a very enjoyable read!
I’m having trouble debugging a multi-process application (specifically using a process pool in python’s multiprocessing module). I have an apparent deadlock and I do not know what is causing it. The stack trace is not sufficient to describe the issue, as it only displays code in the multiprocessing module.
Are there any python tools, or otherwise general techniques used to debug deadlocks?
Yah, debugging deadlocks is fun. You can set the logging level to be higher — see the Python documentation for a description of it, but really quickly:
import multiprocessing, logging
logger = multiprocessing.log_to_stderr()
logger.setLevel(multiprocessing.SUBDEBUG)
Also, add logging for anything in your code that deals with a resource or whatnot that might be in contention. Finally, shot in the dark: spawning off child processes during an import might cause a problem.
In order to avoid deadlocks in the first place, learning good practices is useful, as parallel processing is indeed quite subtle. The (free) Little Book of Semaphores can be a very enjoyable read!