How to debug a Python segmentation fault?
Question:
How can I debug a Python segmentation fault?
We are trying to run our python code on SuSE 12.3. We get reproducible segmentation faults. The python code has been working on other platforms without segmentation faults, for years.
We only code Python, no C extension ….
What is the best way to debug this? I know a bit ansi c, but that was ten years ago ….
Python 2.7.5
Update
The segmentation fault happens on interpreter shutdown.
I can run the script several times:
python -m pdb myscript.py arg1 arg1
continue
run
continue
run
But the segmentation faults happen, if I leave the pdb with ctrl-d.
Update 2
I now try to debug it with gdb:
gdb
> file python
> run myscript.py arg1 arg2
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffefbe2700 (LWP 15483)]
0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
(gdb) bt
#0 0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#1 0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#2 0x00007ffff7adc858 in ?? () from /usr/lib64/libpython2.7.so.1.0
#3 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#4 0x00007ffff7af1082 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#5 0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#6 0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#7 0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#8 0x00007ffff7adc5b6 in ?? () from /usr/lib64/libpython2.7.so.1.0
#9 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#10 0x00007ffff7ad9171 in ?? () from /usr/lib64/libpython2.7.so.1.0
#11 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#12 0x00007ffff7aeeb62 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.7.so.1.0
#13 0x00007ffff7acc757 in ?? () from /usr/lib64/libpython2.7.so.1.0
#14 0x00007ffff7828e0f in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff755c7dd in clone () from /lib64/libc.so.6
Update 3
I installed gdbinit from http://hg.python.org/cpython/file/default/Misc/gdbinit
and the debugging symbols from http://download.opensuse.org/debug/distribution/12.3/repo/oss/suse/x86_64/
(gdb) pystack
No symbol "_PyUnicode_AsString" in current context.
What now?
Update 4
We installed the a new RPM (python-2.7.5-3.1.x86_64). We get less segfaults, but they still happen.
Here is the link to repository:
http://download.opensuse.org/repositories/devel:/languages:/python:/Factory/openSUSE_12.3/x86_64/
Update 5
Solved my initial problem:
It was http://bugs.python.org/issue1856 (shutdown (exit) can hang or segfault with daemon threads running)
Answers:
If you’re executing nothing but Python code (even through your imported third party modules), than a segfault probably means there’s a bug in the interpreter or one of its builtin C modules.
You can either build CPython and try to debug it yourself, or try to produce the smallest script which reproduces the crash and file an issue.
Maybe there is a daemon thread running? There is a reproduceable bug, which was fixed only for 3.x, but not for 2.x:
http://bugs.python.org/issue1856:
shutdown (exit) can hang or segfault with daemon threads running
This is the answer to my own question. It took some time to find the root of the problem.
Here is the next question: How to code around this bug: Detect Interpreter shut down in daemon thread
I got to this question because of the Segmentation fault, but not on exit, just in general, and I found that nothing else helped as effectively as faulthandler. It’s part of Python 3.3, and you can install in 2.7 using pip.
tl;dr for python3 users.
Firstly, from the docs:
faulthandler is a builtin module since Python 3.3
Code usage:
import faulthandler
faulthandler.enable()
// bad code goes here
Shell usage:
$ python3 -q -X faulthandler
>>> /// bad cod goes here
You can do this with faulthandler as mentioned. E.g.
import faulthandler; faulthandler.enable()
Just add this line near your import statement and run the code. It will help to debug you or will try to show you nearest line in your code which caused the segmentation fault. Then you can make changes wherever required.
How can I debug a Python segmentation fault?
We are trying to run our python code on SuSE 12.3. We get reproducible segmentation faults. The python code has been working on other platforms without segmentation faults, for years.
We only code Python, no C extension ….
What is the best way to debug this? I know a bit ansi c, but that was ten years ago ….
Python 2.7.5
Update
The segmentation fault happens on interpreter shutdown.
I can run the script several times:
python -m pdb myscript.py arg1 arg1
continue
run
continue
run
But the segmentation faults happen, if I leave the pdb with ctrl-d.
Update 2
I now try to debug it with gdb:
gdb
> file python
> run myscript.py arg1 arg2
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffefbe2700 (LWP 15483)]
0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
(gdb) bt
#0 0x00007ffff7aef93c in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#1 0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#2 0x00007ffff7adc858 in ?? () from /usr/lib64/libpython2.7.so.1.0
#3 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#4 0x00007ffff7af1082 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#5 0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#6 0x00007ffff7af233d in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0
#7 0x00007ffff7af5303 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0
#8 0x00007ffff7adc5b6 in ?? () from /usr/lib64/libpython2.7.so.1.0
#9 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#10 0x00007ffff7ad9171 in ?? () from /usr/lib64/libpython2.7.so.1.0
#11 0x00007ffff7ad840d in PyObject_Call () from /usr/lib64/libpython2.7.so.1.0
#12 0x00007ffff7aeeb62 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.7.so.1.0
#13 0x00007ffff7acc757 in ?? () from /usr/lib64/libpython2.7.so.1.0
#14 0x00007ffff7828e0f in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff755c7dd in clone () from /lib64/libc.so.6
Update 3
I installed gdbinit from http://hg.python.org/cpython/file/default/Misc/gdbinit
and the debugging symbols from http://download.opensuse.org/debug/distribution/12.3/repo/oss/suse/x86_64/
(gdb) pystack
No symbol "_PyUnicode_AsString" in current context.
What now?
Update 4
We installed the a new RPM (python-2.7.5-3.1.x86_64). We get less segfaults, but they still happen.
Here is the link to repository:
http://download.opensuse.org/repositories/devel:/languages:/python:/Factory/openSUSE_12.3/x86_64/
Update 5
Solved my initial problem:
It was http://bugs.python.org/issue1856 (shutdown (exit) can hang or segfault with daemon threads running)
If you’re executing nothing but Python code (even through your imported third party modules), than a segfault probably means there’s a bug in the interpreter or one of its builtin C modules.
You can either build CPython and try to debug it yourself, or try to produce the smallest script which reproduces the crash and file an issue.
Maybe there is a daemon thread running? There is a reproduceable bug, which was fixed only for 3.x, but not for 2.x:
http://bugs.python.org/issue1856:
shutdown (exit) can hang or segfault with daemon threads running
This is the answer to my own question. It took some time to find the root of the problem.
Here is the next question: How to code around this bug: Detect Interpreter shut down in daemon thread
I got to this question because of the Segmentation fault, but not on exit, just in general, and I found that nothing else helped as effectively as faulthandler. It’s part of Python 3.3, and you can install in 2.7 using pip.
tl;dr for python3 users.
Firstly, from the docs:
faulthandler is a builtin module since Python 3.3
Code usage:
import faulthandler
faulthandler.enable()
// bad code goes here
Shell usage:
$ python3 -q -X faulthandler
>>> /// bad cod goes here
You can do this with faulthandler as mentioned. E.g.
import faulthandler; faulthandler.enable()
Just add this line near your import statement and run the code. It will help to debug you or will try to show you nearest line in your code which caused the segmentation fault. Then you can make changes wherever required.