How to use valgrind with python?
Question:
I am trying to memcheck a C python extension I am writing, but I’m having trouble setting up valgrind to work with python. I would really appreciate some advice. Just for context, this is Ubuntu 13.10, python 2.7.5+, and valgrind 3.8.1.
As per recommendation from Readme.valgrind I did the following.
1) Downloaded the python source with
sudo apt-get build-dep python2.7
apt-get source python2.7
2) Applied the code patch, i.e. “Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c”.
3) Applied the suppression patch, i.e. “Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc”
4) Compiled python with
./configure --prefix=/home/dejan/workspace/python --without-pymalloc
make -j4 install
Note that I did both 2 and 3, while README.valgrind says to do 2 or 3… more can’t hurt.
Now, let’s test this on some sample python code in test.py
print "Test"
Let’s run valgrind on python with this script
valgrind --tool=memcheck --leak-check=full --suppressions=python2.7-2.7.5/Misc/valgrind-python.supp bin/python test.py
Unexpectedly, there is still loads of reports from valgrind, with the first one being (and many more following)
==27944== HEAP SUMMARY:
==27944== in use at exit: 857,932 bytes in 5,144 blocks
==27944== total heap usage: 22,766 allocs, 17,622 frees, 4,276,934 bytes allocated
==27944==
==27944== 38 bytes in 1 blocks are possibly lost in loss record 24 of 1,343
==27944== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27944== by 0x46B8DD: PyString_FromString (stringobject.c:143)
==27944== by 0x439631: PyFile_FromFile (fileobject.c:157)
==27944== by 0x4E9B4A: _PySys_Init (sysmodule.c:1383)
==27944== by 0x4E29E9: Py_InitializeEx (pythonrun.c:222)
==27944== by 0x4154B4: Py_Main (main.c:546)
==27944== by 0x577DDE4: (below main) (libc-start.c:260)
Am I doing something wrong? Is there a way to valgrind a python script that doesn’t leak and get clean valgrind output?
Answers:
I found the answer here.
Python also needs to be compiled in debug mode, i.e.
./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind
In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.
Since python 3.6, there’s a PYTHONMALLOC environment variable which is available in release builds, without needing to recompile.
PYTHONMALLOC=malloc python3 foobar.py
This will disable pymalloc and just use the libc malloc directly, making it valgrind-friendly. This is equivalent to --without-pymalloc (and it is just as slow)
If valgrind is too slow, other values can be helpful. PYTHONMALLOC=debug and PYTHONMALLOC=malloc_debug add debug hooks on top of the default and the libc allocators respectively. Their effects, from the docs:
- Newly allocated memory is filled with the byte 0xCB
- Freed memory is filled with the byte 0xDB
- Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
- Detect writes before the start of a buffer (buffer underflows)
- Detect writes after the end of a buffer (buffer overflows)
- Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.
This will catch some uninitialized reads, some use after free, some buffer under/overflows, etc, but won’t report leaks and won’t touch memory that isn’t allocated through python (When using glibc, the MALLOC_PERTURB_ and MALLOC_CHECK_ environment variables might help there)
See also:
I am trying to memcheck a C python extension I am writing, but I’m having trouble setting up valgrind to work with python. I would really appreciate some advice. Just for context, this is Ubuntu 13.10, python 2.7.5+, and valgrind 3.8.1.
As per recommendation from Readme.valgrind I did the following.
1) Downloaded the python source with
sudo apt-get build-dep python2.7
apt-get source python2.7
2) Applied the code patch, i.e. “Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c”.
3) Applied the suppression patch, i.e. “Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc”
4) Compiled python with
./configure --prefix=/home/dejan/workspace/python --without-pymalloc
make -j4 install
Note that I did both 2 and 3, while README.valgrind says to do 2 or 3… more can’t hurt.
Now, let’s test this on some sample python code in test.py
print "Test"
Let’s run valgrind on python with this script
valgrind --tool=memcheck --leak-check=full --suppressions=python2.7-2.7.5/Misc/valgrind-python.supp bin/python test.py
Unexpectedly, there is still loads of reports from valgrind, with the first one being (and many more following)
==27944== HEAP SUMMARY:
==27944== in use at exit: 857,932 bytes in 5,144 blocks
==27944== total heap usage: 22,766 allocs, 17,622 frees, 4,276,934 bytes allocated
==27944==
==27944== 38 bytes in 1 blocks are possibly lost in loss record 24 of 1,343
==27944== at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27944== by 0x46B8DD: PyString_FromString (stringobject.c:143)
==27944== by 0x439631: PyFile_FromFile (fileobject.c:157)
==27944== by 0x4E9B4A: _PySys_Init (sysmodule.c:1383)
==27944== by 0x4E29E9: Py_InitializeEx (pythonrun.c:222)
==27944== by 0x4154B4: Py_Main (main.c:546)
==27944== by 0x577DDE4: (below main) (libc-start.c:260)
Am I doing something wrong? Is there a way to valgrind a python script that doesn’t leak and get clean valgrind output?
I found the answer here.
Python also needs to be compiled in debug mode, i.e.
./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind
In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.
Since python 3.6, there’s a PYTHONMALLOC environment variable which is available in release builds, without needing to recompile.
PYTHONMALLOC=malloc python3 foobar.py
This will disable pymalloc and just use the libc malloc directly, making it valgrind-friendly. This is equivalent to --without-pymalloc (and it is just as slow)
If valgrind is too slow, other values can be helpful. PYTHONMALLOC=debug and PYTHONMALLOC=malloc_debug add debug hooks on top of the default and the libc allocators respectively. Their effects, from the docs:
- Newly allocated memory is filled with the byte 0xCB
- Freed memory is filled with the byte 0xDB
- Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
- Detect writes before the start of a buffer (buffer underflows)
- Detect writes after the end of a buffer (buffer overflows)
- Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.
This will catch some uninitialized reads, some use after free, some buffer under/overflows, etc, but won’t report leaks and won’t touch memory that isn’t allocated through python (When using glibc, the MALLOC_PERTURB_ and MALLOC_CHECK_ environment variables might help there)
See also: