Embedding Python interpreter in C leads to segfault when loading with ctypes
Question:
I try to embed the Python Interpreter into C.
In order to test this I create a shared library and
try to load this one in Python with ctypes. Unfortunately this doesn’t
work and I would like to understand why.
Here is an example c – code:
#ifdef __cplusplus
extern "C" {
#endif
#include <Python.h>
int run_py(void);
int run_py2(void);
int
run_py(void)
{
printf("hello from run_pyn");
return 42;
}
int
run_py2(void)
{
printf("entering c-function: run_py()n");
Py_Initialize();
PyRun_SimpleString("print('hello world')");
return 0;
}
#ifdef __cplusplus
}
#endif
So I compile this into “mylib.so” with gcc and use python3.7-config –cflags and –ldflags for linking and so on which works.
Here is the Python code I use to load this..
import ctypes as c
import os
import sys
if __name__ == '__main__':
print("running shared-lib integration test with python:n{}".format(sys.version))
path = os.path.dirname(os.path.realpath(__file__))
dllfile = os.path.join(path, 'mylib.so')
dll = c.CDLL(str(dllfile))
print("loaded CDLL")
dll.run_py.restype = c.c_int
dll.run_py2.restype = c.c_int
print("now calling dll.run_py()...")
rv = dll.run_py()
print("called dll.run_py: rv={}".format(rv))
print("now calling dll.run_py2()...")
rv2 = dll.run_py2()
print("called dll.run_py2: rv={}".format(rv2))
So this simply loads both functions run_py and run_py2
and executes them. This is the output…
running shared-lib integration test with python:
3.7.1 (default, Oct 22 2018, 10:41:28)
[GCC 8.2.1 20180831]
loaded CDLL
now calling dll.run_py()...
hello from run_py
called dll.run_py: rv=42
now calling dll.run_py2()...
entering c-function: run_py()
Segmentation fault (core dumped)
So basically this leads to segfault when calling run_py2.
The cause for this is the call of PyRun_SimpleString .
However if I compile this as a standalone C programm
everything seems to work just fine. I really
would like to understand why this happens… but currently im
out ouf ideas so any feedback is really appreciated here.
BR jrsm
Answers:
I changed your code a bit. Also, I’m testing on Win (as it’s more convenient for me at this point), but I’m sure things are the same in Nix.
dll00.c:
#include <stdio.h>
#include <Python.h>
#define PRINT_MSG_0() printf("From C - [%s] (%d) - [%s]n", __FILE__, __LINE__, __FUNCTION__)
#if defined(_WIN32)
# define DLL_EXPORT_API __declspec(dllexport)
#else
# define DLL_EXPORT_API
#endif
#if defined(__cplusplus)
extern "C" {
#endif
DLL_EXPORT_API int test0();
DLL_EXPORT_API int test1();
#if defined(__cplusplus)
}
#endif
int test0()
{
PRINT_MSG_0();
return 42;
}
int test1()
{
PRINT_MSG_0();
Py_Initialize();
PRINT_MSG_0();
PyRun_SimpleString("print("Hello world!!!")");
PRINT_MSG_0();
return 0;
}
code00.py:
#!/usr/bin/env python
import ctypes as cts
import sys
DLL_NAME = "./dll00.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
def main():
dll = cts.PyDLL(DLL_NAME)
test0 = dll.test0
test0.argtypes = None
test0.restype = cts.c_int
test1 = dll.test1
test1.argtypes = None
test1.restype = cts.c_int
print("Calling {:}...".format(test0.__name__))
res = test0()
print("{:} returned {:d}".format(test0.__name__, res))
print("Calling {:}...".format(test1.__name__))
res = test1()
print("{:} returned {:d}".format(test1.__name__, res))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.n")
sys.exit(rc)
Output:
(py35x64_test) e:WorkDevStackOverflowq053609932> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[prompt]> "c:Installx86MicrosoftVisual Studio Community2015vcvcvarsall.bat" x64
[prompt]> dir /b
code00.py
dll00.c
[prompt]> cl /nologo /DDLL /MD /Ic:Installx64PythonPython3.5include dll00.c /link /NOLOGO /DLL /OUT:dll00.so /LIBPATH:c:Installx64PythonPython3.5libs
dll00.c
Creating library dll00.lib and object dll00.exp
[prompt]> dir /b
code00.py
dll00.c
dll00.exp
dll00.lib
dll00.obj
dll00.so
[prompt]> "e:WorkDevVEnvspy35x64_testScriptspython.exe" code00.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
Calling test0...
From C - [dll00.c] (26) - [test0]
test0 returned 42
Calling test1...
From C - [dll00.c] (32) - [test1]
From C - [dll00.c] (34) - [test1]
Traceback (most recent call last):
File "code00.py", line 30, in <module>
main()
File "code00.py", line 24, in main
res = test1_func()
OSError: exception: access violation reading 0x0000000000000010
The problem reproduces. First, I thought it’s the [Python.Docs]: Initialization, Finalization, and Threads – void Py_Initialize() call. But then I remembered [Python.Docs]: class ctypes.PyDLL(name, mode=DEFAULT_MODE, handle=None) which states (emphasis is mine):
Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.
Thus, this is only useful to call Python C api functions directly.
Replacing CDLL by PyDLL in code00.py, yields:
[prompt]> "e:WorkDevVEnvspy35x64_testScriptspython.exe" code00.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
Calling test0...
From C - [dll00.c] (26) - [test0]
test0 returned 42
Calling test1...
From C - [dll00.c] (32) - [test1]
From C - [dll00.c] (34) - [test1]
Hello world!!!
From C - [dll00.c] (36) - [test1]
test1 returned 0
Output (Nix with PyDLL):
(py_pc064_03.08_test0_lancer) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackExchange/StackOverflow/q053609932]> . ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[064bit prompt]> gcc -fPIC -shared -o dll00.so dll00.c -I/usr/include/python3.8
[064bit prompt]> python ./code00.py
Python 3.8.18 (default, Aug 25 2023, 13:20:30) [GCC 11.4.0] 064bit on linux
Calling test0...
From C - [dll00.c] (27) - [test0]
test0 returned 42
Calling test1...
From C - [dll00.c] (34) - [test1]
From C - [dll00.c] (36) - [test1]
Hello world!!!
From C - [dll00.c] (38) - [test1]
test1 returned 0
Done.
Might be related:
I try to embed the Python Interpreter into C.
In order to test this I create a shared library and
try to load this one in Python with ctypes. Unfortunately this doesn’t
work and I would like to understand why.
Here is an example c – code:
#ifdef __cplusplus
extern "C" {
#endif
#include <Python.h>
int run_py(void);
int run_py2(void);
int
run_py(void)
{
printf("hello from run_pyn");
return 42;
}
int
run_py2(void)
{
printf("entering c-function: run_py()n");
Py_Initialize();
PyRun_SimpleString("print('hello world')");
return 0;
}
#ifdef __cplusplus
}
#endif
So I compile this into “mylib.so” with gcc and use python3.7-config –cflags and –ldflags for linking and so on which works.
Here is the Python code I use to load this..
import ctypes as c
import os
import sys
if __name__ == '__main__':
print("running shared-lib integration test with python:n{}".format(sys.version))
path = os.path.dirname(os.path.realpath(__file__))
dllfile = os.path.join(path, 'mylib.so')
dll = c.CDLL(str(dllfile))
print("loaded CDLL")
dll.run_py.restype = c.c_int
dll.run_py2.restype = c.c_int
print("now calling dll.run_py()...")
rv = dll.run_py()
print("called dll.run_py: rv={}".format(rv))
print("now calling dll.run_py2()...")
rv2 = dll.run_py2()
print("called dll.run_py2: rv={}".format(rv2))
So this simply loads both functions run_py and run_py2
and executes them. This is the output…
running shared-lib integration test with python:
3.7.1 (default, Oct 22 2018, 10:41:28)
[GCC 8.2.1 20180831]
loaded CDLL
now calling dll.run_py()...
hello from run_py
called dll.run_py: rv=42
now calling dll.run_py2()...
entering c-function: run_py()
Segmentation fault (core dumped)
So basically this leads to segfault when calling run_py2.
The cause for this is the call of PyRun_SimpleString .
However if I compile this as a standalone C programm
everything seems to work just fine. I really
would like to understand why this happens… but currently im
out ouf ideas so any feedback is really appreciated here.
BR jrsm
I changed your code a bit. Also, I’m testing on Win (as it’s more convenient for me at this point), but I’m sure things are the same in Nix.
dll00.c:
#include <stdio.h>
#include <Python.h>
#define PRINT_MSG_0() printf("From C - [%s] (%d) - [%s]n", __FILE__, __LINE__, __FUNCTION__)
#if defined(_WIN32)
# define DLL_EXPORT_API __declspec(dllexport)
#else
# define DLL_EXPORT_API
#endif
#if defined(__cplusplus)
extern "C" {
#endif
DLL_EXPORT_API int test0();
DLL_EXPORT_API int test1();
#if defined(__cplusplus)
}
#endif
int test0()
{
PRINT_MSG_0();
return 42;
}
int test1()
{
PRINT_MSG_0();
Py_Initialize();
PRINT_MSG_0();
PyRun_SimpleString("print("Hello world!!!")");
PRINT_MSG_0();
return 0;
}
code00.py:
#!/usr/bin/env python
import ctypes as cts
import sys
DLL_NAME = "./dll00.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
def main():
dll = cts.PyDLL(DLL_NAME)
test0 = dll.test0
test0.argtypes = None
test0.restype = cts.c_int
test1 = dll.test1
test1.argtypes = None
test1.restype = cts.c_int
print("Calling {:}...".format(test0.__name__))
res = test0()
print("{:} returned {:d}".format(test0.__name__, res))
print("Calling {:}...".format(test1.__name__))
res = test1()
print("{:} returned {:d}".format(test1.__name__, res))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.n")
sys.exit(rc)
Output:
(py35x64_test) e:WorkDevStackOverflowq053609932> sopr.bat ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [prompt]> "c:Installx86MicrosoftVisual Studio Community2015vcvcvarsall.bat" x64 [prompt]> dir /b code00.py dll00.c [prompt]> cl /nologo /DDLL /MD /Ic:Installx64PythonPython3.5include dll00.c /link /NOLOGO /DLL /OUT:dll00.so /LIBPATH:c:Installx64PythonPython3.5libs dll00.c Creating library dll00.lib and object dll00.exp [prompt]> dir /b code00.py dll00.c dll00.exp dll00.lib dll00.obj dll00.so [prompt]> "e:WorkDevVEnvspy35x64_testScriptspython.exe" code00.py Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32 Calling test0... From C - [dll00.c] (26) - [test0] test0 returned 42 Calling test1... From C - [dll00.c] (32) - [test1] From C - [dll00.c] (34) - [test1] Traceback (most recent call last): File "code00.py", line 30, in <module> main() File "code00.py", line 24, in main res = test1_func() OSError: exception: access violation reading 0x0000000000000010
The problem reproduces. First, I thought it’s the [Python.Docs]: Initialization, Finalization, and Threads – void Py_Initialize() call. But then I remembered [Python.Docs]: class ctypes.PyDLL(name, mode=DEFAULT_MODE, handle=None) which states (emphasis is mine):
Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.
Thus, this is only useful to call Python C api functions directly.
Replacing CDLL by PyDLL in code00.py, yields:
[prompt]> "e:WorkDevVEnvspy35x64_testScriptspython.exe" code00.py Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32 Calling test0... From C - [dll00.c] (26) - [test0] test0 returned 42 Calling test1... From C - [dll00.c] (32) - [test1] From C - [dll00.c] (34) - [test1] Hello world!!! From C - [dll00.c] (36) - [test1] test1 returned 0
Output (Nix with PyDLL):
(py_pc064_03.08_test0_lancer) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackExchange/StackOverflow/q053609932]> . ~/sopr.sh ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [064bit prompt]> gcc -fPIC -shared -o dll00.so dll00.c -I/usr/include/python3.8 [064bit prompt]> python ./code00.py Python 3.8.18 (default, Aug 25 2023, 13:20:30) [GCC 11.4.0] 064bit on linux Calling test0... From C - [dll00.c] (27) - [test0] test0 returned 42 Calling test1... From C - [dll00.c] (34) - [test1] From C - [dll00.c] (36) - [test1] Hello world!!! From C - [dll00.c] (38) - [test1] test1 returned 0 Done.
Might be related: