python tracing a segmentation fault
Question:
I’m developing C extensions from python and I obtain some segfaults (inevitable during the development…).
I’m searching for a way to display at which line of code the segfault happens (an idea is like tracing every single line of code), how can I do that?
Answers:
Here’s a way to output the filename and line number of every line of Python your code runs:
import sys
def trace(frame, event, arg):
print("%s, %s:%d" % (event, frame.f_code.co_filename, frame.f_lineno))
return trace
def test():
print("Line 8")
print("Line 9")
sys.settrace(trace)
test()
Output:
call, test.py:7
line, test.py:8
Line 8
line, test.py:9
Line 9
return, test.py:9
(You’d probably want to write the trace output to a file, of course.)
If you are on linux, run python under gdb
gdb python
(gdb) run /path/to/script.py
## wait for segfault ##
(gdb) backtrace
## stack trace of the c code
Segfaults from C extensions are very frequently a result of not incrementing a reference count when you create a new reference to an object. That makes them very hard to track down as the segfault occurs only after the last reference is removed from the object, and even then often only when some other object is being allocated.
You don’t say how much C extension code you have written so far, but if you’re just starting out consider whether you can use either ctypes or Cython. Ctypes may not be flexible enough for your needs, but you should be able to link to just about any C library with Cython and have all the reference counts maintained for you automatically.
That isn’t always sufficient: if your Python objects and any underlying C objects have different lifetimes you can still get problems, but it does simplify things considerably.
There are somewhat undocumented python extensions for gdb.
From the Python source grab Tools/gdb/libpython.py (it is not included in a normal install).
Put this in sys.path
Then:
# gdb /gps/python2.7_x64/bin/python coredump
...
Core was generated by `/usr/bin/python script.py'.
Program terminated with signal 11, Segmentation fault.
#0 call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
...
(gdb) python
>import libpython
>
>end
(gdb) bt
#0 call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
#1 PyEval_EvalFrameEx (f=f@entry=
Frame 0x7f9084d20ad0,
for file /usr/lib/python2.7/site-packages/librabbitmq/__init__.py, line 220,
in drain_events (self=<Connection(channels={1: <Channel(channel_id=1, connection=<...>, is_open=True, connect_timeout=4, _default_channel=<....(truncated), throwflag=throwflag@entry=0) at Python/ceval.c:2681
...
(gdb) py-list
218 else:
219 timeout = float(timeout)
>220 self._basic_recv(timeout)
221
222 def channel(self, channel_id=None):
As you can see we now have visibility into the Python stack corresponding with the CPython call chain.
Some caveats:
- Your version of gdb needs to be greater than 7 and it needs to have been compiled with
--with-python
gdb embeds python (by linking to libpython), it doesn’t run it in a subshell. This means that It may not necessarily match the version of python that is on $PATH. - You need to download
libpython.py from whatever version of the Python source that matches whatever gdb is linked to.
- You may have to run gdb as root – if so you may need to set up
sys.path to match that of the code that you are debugging.
If you cannot copy libpython.py into sys.path then you can add it’s location to sys.path like this:
(gdb) python
>import sys
>sys.path.append('/path/to/containing/dir/')
>import libpython
>
>end
This is somewhat poorly documented in the python dev docs, the fedora wiki and the python wiki
If you have an older gdb or just can’t get this working there is also a gdbinit in the Python source that you can copy to ~/.gdbinit which add some similar functionality
I came here looking for a solution to the same problem, and none of the other answers helped me. What did help was faulthandler, and you can install it in Python 2.7 just using pip install.
faulthandler was introduced to Python only in version 3.3, that was released in September 2012, which was after most other answers here were written.
Here are 3 more alternatives:
1: Executing a script with faulthandler enabled:
python3 -X faulthandler your_script.py
2: Executing a script in debug mode (pdb)
python3 -m pdb your_script.py
and execute the script with the continue command.
The gdb tool provided the most information, but none of them print the last executed line number in my script.
3: I ended up using pytest. For this to work, I wrapped my code in a function prefixed with test_ and execute the script like this:
pytest your_script.py
Mark’s answer is awesome. If you happen to be on a machine that has lldb readily available in your path, and not gdb (which was my case), Mark’s answer becomes:
lldb python
(lldb) process launch -- /path/to/script.py
## wait for segfault ##
(lldb) bt
## stack trace of the c code
I wish I would have stumbled on this answer earlier 🙂
I’m developing C extensions from python and I obtain some segfaults (inevitable during the development…).
I’m searching for a way to display at which line of code the segfault happens (an idea is like tracing every single line of code), how can I do that?
Here’s a way to output the filename and line number of every line of Python your code runs:
import sys
def trace(frame, event, arg):
print("%s, %s:%d" % (event, frame.f_code.co_filename, frame.f_lineno))
return trace
def test():
print("Line 8")
print("Line 9")
sys.settrace(trace)
test()
Output:
call, test.py:7
line, test.py:8
Line 8
line, test.py:9
Line 9
return, test.py:9
(You’d probably want to write the trace output to a file, of course.)
If you are on linux, run python under gdb
gdb python
(gdb) run /path/to/script.py
## wait for segfault ##
(gdb) backtrace
## stack trace of the c code
Segfaults from C extensions are very frequently a result of not incrementing a reference count when you create a new reference to an object. That makes them very hard to track down as the segfault occurs only after the last reference is removed from the object, and even then often only when some other object is being allocated.
You don’t say how much C extension code you have written so far, but if you’re just starting out consider whether you can use either ctypes or Cython. Ctypes may not be flexible enough for your needs, but you should be able to link to just about any C library with Cython and have all the reference counts maintained for you automatically.
That isn’t always sufficient: if your Python objects and any underlying C objects have different lifetimes you can still get problems, but it does simplify things considerably.
There are somewhat undocumented python extensions for gdb.
From the Python source grab Tools/gdb/libpython.py (it is not included in a normal install).
Put this in sys.path
Then:
# gdb /gps/python2.7_x64/bin/python coredump
...
Core was generated by `/usr/bin/python script.py'.
Program terminated with signal 11, Segmentation fault.
#0 call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
...
(gdb) python
>import libpython
>
>end
(gdb) bt
#0 call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
#1 PyEval_EvalFrameEx (f=f@entry=
Frame 0x7f9084d20ad0,
for file /usr/lib/python2.7/site-packages/librabbitmq/__init__.py, line 220,
in drain_events (self=<Connection(channels={1: <Channel(channel_id=1, connection=<...>, is_open=True, connect_timeout=4, _default_channel=<....(truncated), throwflag=throwflag@entry=0) at Python/ceval.c:2681
...
(gdb) py-list
218 else:
219 timeout = float(timeout)
>220 self._basic_recv(timeout)
221
222 def channel(self, channel_id=None):
As you can see we now have visibility into the Python stack corresponding with the CPython call chain.
Some caveats:
- Your version of gdb needs to be greater than 7 and it needs to have been compiled with
--with-python gdbembeds python (by linking tolibpython), it doesn’t run it in a subshell. This means that It may not necessarily match the version of python that is on$PATH.- You need to download
libpython.pyfrom whatever version of the Python source that matches whatevergdbis linked to. - You may have to run gdb as root – if so you may need to set up
sys.pathto match that of the code that you are debugging.
If you cannot copy libpython.py into sys.path then you can add it’s location to sys.path like this:
(gdb) python
>import sys
>sys.path.append('/path/to/containing/dir/')
>import libpython
>
>end
This is somewhat poorly documented in the python dev docs, the fedora wiki and the python wiki
If you have an older gdb or just can’t get this working there is also a gdbinit in the Python source that you can copy to ~/.gdbinit which add some similar functionality
I came here looking for a solution to the same problem, and none of the other answers helped me. What did help was faulthandler, and you can install it in Python 2.7 just using pip install.
faulthandler was introduced to Python only in version 3.3, that was released in September 2012, which was after most other answers here were written.
Here are 3 more alternatives:
1: Executing a script with faulthandler enabled:
python3 -X faulthandler your_script.py
2: Executing a script in debug mode (pdb)
python3 -m pdb your_script.py
and execute the script with the continue command.
The gdb tool provided the most information, but none of them print the last executed line number in my script.
3: I ended up using pytest. For this to work, I wrapped my code in a function prefixed with test_ and execute the script like this:
pytest your_script.py
Mark’s answer is awesome. If you happen to be on a machine that has lldb readily available in your path, and not gdb (which was my case), Mark’s answer becomes:
lldb python
(lldb) process launch -- /path/to/script.py
## wait for segfault ##
(lldb) bt
## stack trace of the c code
I wish I would have stumbled on this answer earlier 🙂