Dump stacktraces of all active Threads
Question:
I’m trying to dump a list of all active threads including the current stack of each. I can get a list of all threads using threading.enumerate(), but i can’t figure out a way to get to the stack from there.
Background: A Zope/Plone app freaks out from time to time, consuming 100% of cpu and needs to be restarted. I have a feeling it’s a loop which doesn’t terminate properly, but i cannot reproduce it in the test-environemt for verification. I managed to register a signal handler which can be triggered from the outside, so i can trigger some code as soon as the situation occurs again. If I could dump the stacktrace for all active threads, that would give me a clue what goes wrong. The hole thing runs on python 2.4…
Any ideas on how to trace down situations like these are appreciated 🙂
Cheers,
Chriss
Answers:
There is an applicable recipe on ASPN. You can use threading.enumerate() to get all the tids, then just call _async_raise() with some suitable exception to force a stack trace.
2.4. Too bad. From Python 2.5 on there is sys._current_frames().
But you could try threadframe. And if the makefile gives you trouble you could try this setup.py for threadframe
When using Zope, you want to install Products.signalstack or mr.freeze; these were designed for just this purpose!
Send a USR1 signal to your Zope server and it’ll immediately dump stack traces for all threads to the console. It’ll do this even if all Zope threads are locked up.
Under the hood these packages indirectly use threadframes; for Python versions 2.5 and up, when not using Zope, you can build the same functionality using the sys._current_frames() function to access per-thread stack frames.
As of Zope 2.12.5 this functionality is integrated into Zope itself, and there is no need to install additional packages anymore.
As jitter points out in an earlier answer sys._current_frames() gives you what you need for v2.5+. For the lazy the following code snippet worked for me and may help you:
print >> sys.stderr, "n*** STACKTRACE - START ***n"
code = []
for threadId, stack in sys._current_frames().items():
code.append("n# ThreadID: %s" % threadId)
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename,
lineno, name))
if line:
code.append(" %s" % (line.strip()))
for line in code:
print >> sys.stderr, line
print >> sys.stderr, "n*** STACKTRACE - END ***n"
For Python 3.3 and later, there is faulthandler.dump_traceback().
The code below produces similar output, but includes the thread name and could be enhanced to print more information.
for th in threading.enumerate():
print(th)
traceback.print_stack(sys._current_frames()[th.ident])
print()
Just for completeness sake, Products.LongRequestLogger is super helpful to identify bottlenecks, and to do so it dumps stacktraces at specific intervals.
I’m trying to dump a list of all active threads including the current stack of each. I can get a list of all threads using threading.enumerate(), but i can’t figure out a way to get to the stack from there.
Background: A Zope/Plone app freaks out from time to time, consuming 100% of cpu and needs to be restarted. I have a feeling it’s a loop which doesn’t terminate properly, but i cannot reproduce it in the test-environemt for verification. I managed to register a signal handler which can be triggered from the outside, so i can trigger some code as soon as the situation occurs again. If I could dump the stacktrace for all active threads, that would give me a clue what goes wrong. The hole thing runs on python 2.4…
Any ideas on how to trace down situations like these are appreciated 🙂
Cheers,
Chriss
There is an applicable recipe on ASPN. You can use threading.enumerate() to get all the tids, then just call _async_raise() with some suitable exception to force a stack trace.
2.4. Too bad. From Python 2.5 on there is sys._current_frames().
But you could try threadframe. And if the makefile gives you trouble you could try this setup.py for threadframe
When using Zope, you want to install Products.signalstack or mr.freeze; these were designed for just this purpose!
Send a USR1 signal to your Zope server and it’ll immediately dump stack traces for all threads to the console. It’ll do this even if all Zope threads are locked up.
Under the hood these packages indirectly use threadframes; for Python versions 2.5 and up, when not using Zope, you can build the same functionality using the sys._current_frames() function to access per-thread stack frames.
As of Zope 2.12.5 this functionality is integrated into Zope itself, and there is no need to install additional packages anymore.
As jitter points out in an earlier answer sys._current_frames() gives you what you need for v2.5+. For the lazy the following code snippet worked for me and may help you:
print >> sys.stderr, "n*** STACKTRACE - START ***n"
code = []
for threadId, stack in sys._current_frames().items():
code.append("n# ThreadID: %s" % threadId)
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename,
lineno, name))
if line:
code.append(" %s" % (line.strip()))
for line in code:
print >> sys.stderr, line
print >> sys.stderr, "n*** STACKTRACE - END ***n"
For Python 3.3 and later, there is faulthandler.dump_traceback().
The code below produces similar output, but includes the thread name and could be enhanced to print more information.
for th in threading.enumerate():
print(th)
traceback.print_stack(sys._current_frames()[th.ident])
print()
Just for completeness sake, Products.LongRequestLogger is super helpful to identify bottlenecks, and to do so it dumps stacktraces at specific intervals.