Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated
Question:
I wrote a dll library in c, compile with vs2017 64-bit, and try to load it with python3.6 64-bit. However the object’s member variable’s address got truncated to 32-bit.
Here’s my sim.c file, which is compiled to sim.dll:
class Detector {
public:
Detector();
void process(int* pin, int* pout, int n);
private:
int member_var;
};
Detector::Detector()
{
memset(&member_var, 0, sizeof(member_var));
myfile.open("addr_debug.txt");
myfile << "member_var init address: " << &member_var << endl;
}
void Detector::process(int* pin, int* pout, int n);
{
myfile << "member_var process address: " << &member_var << endl;
myfile.close();
}
#define DllExport __declspec( dllexport )
extern "C" {
DllExport Detector* Detector_new() { return new Detector(); }
DllExport void Detector_process(Detector* det, int* pin, int* pout, int n)
{
det->process(pin, pout, n);
}
}
Here’s my python script:
from ctypes import cdll
lib = cdll.LoadLibrary(r'sim.dll')
class Detector(object):
def __init__(self):
self.obj = lib.Detector_new()
def process(self,pin, pout, n):
lib.Detector_process(self.obj,pin, pout, n)
detector = Detector()
n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)
aptr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
bptr = b.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
detector.process(aptr, bptr, n)
Here’s the address of the member_var in addr_debug.txt:
member_var init address: 0000025259E123C4
member_var process address: 0000000059E123C4
So accessing it trigger memory access error:
OSError: exception: access violation reading 0000000059E123C4
Some attempts I tried to understand the issue:
- Define member_var as public instead of private, not help, address still truncated.
- Define member_var as global variable, then the address is ok. So I guess the member_var address truncation happens either when returning the object to python, or passing the object back to dll.
Answers:
Always (CORRECTLY) specify argtypes and restype for functions defined in C, otherwise (C89 style) they will default to int (generally 32bit), generating !!! Undefined Behavior !!!. On 64bit, addresses (larger than 2 GiB) will be truncated (which is exactly what you’re experiencing). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati’s answer) for more details.
Also, when running into issues, don’t forget about [Python.Docs]: ctypes – A foreign function library for Python.
Below it’s an adapted version of your code.
detector.cpp:
#include <stdio.h>
#include <memory.h>
#include <fstream>
#define SIM_EXPORT __declspec(dllexport)
#define C_TAG "From C"
#define PRINT_MSG_3SPI(ARG0, ARG1, ARG2) printf("%s - [%s] (%d) - [%s]: %s: 0x%0p(%d)n", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1, ARG2)
using std::endl;
std::ofstream outFile;
class Detector {
public:
Detector();
void process(int *pIn, int *pOut, int n);
private:
int m_var;
};
Detector::Detector()
: m_var(25) {
outFile.open("addr_debug.txt");
outFile << "m_var init address: " << &m_var << endl;
PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}
void Detector::process(int *pIn, int *pOut, int n)
{
outFile << "m_var process address: " << &m_var << endl;
outFile.close();
PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}
#if defined(__cplusplus)
extern "C" {
#endif
SIM_EXPORT Detector* DetectorNew() { return new Detector(); }
SIM_EXPORT void DetectorProcess(Detector *pDet, int *pIn, int *pOut, int n)
{
pDet->process(pIn, pOut, n);
}
SIM_EXPORT void DetectorDelete(Detector *pDet) { delete pDet; }
#if defined(__cplusplus)
}
#endif
code00.py:
#!/usr/bin/env python
import ctypes as ct
import sys
import numpy as np
IntPtr = ct.POINTER(ct.c_int)
sim_dll = ct.CDLL("./sim.dll")
detector_new_func = sim_dll.DetectorNew
detector_new_func.argtypes = ()
detector_new_func.restype = ct.c_void_p
detector_process_func = sim_dll.DetectorProcess
detector_process_func.argtypes = (ct.c_void_p, IntPtr, IntPtr, ct.c_int)
detector_process_func.restype = None
detector_delete_func = sim_dll.DetectorDelete
detector_delete_func.argtypes = (ct.c_void_p,)
detector_delete_func.restype = None
class Detector():
def __init__(self):
self.obj = detector_new_func()
def process(self, pin, pout, n):
detector_process_func(self.obj, pin, pout, n)
def __del__(self):
detector_delete_func(self.obj)
def main(*argv):
detector = Detector()
n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)
aptr = a.ctypes.data_as(IntPtr)
bptr = b.ctypes.data_as(IntPtr)
detector.process(aptr, bptr, n)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.")
sys.exit(rc)
Notes:
-
As I stated at the beginning, the problem was argtypes and restype not being specified (e.g. for DetectorNew: comment detector_new_func.restype = ct.c_void_p, and you’ll run into the problem again)
-
Code in the question is missing parts (#includes, imports, …), also there are some syntax errors, so it doesn’t compile, and therefore doesn’t follow [SO]: How to create a Minimal, Complete, and Verifiable example (mcve) guidelines. Please when make sure to have MCVE when asking
-
The object that you allocate (new Detector()), must also be deallocated (otherwise, it will generate a memory leak), so I added a function (DetectorDelete – to do that), which is called from (Python) Detector‘s destructor
-
Other (non critical) changes (identifiers renaming, a bit of refactoring, printing to stdout, …)
Output:
(py35x64_tes1) e:WorkDevStackOverflowq052268294>"c:Installx86MicrosoftVisual Studio Community2015vcvcvarsall.bat" x64
(py35x64_test) e:WorkDevStackOverflowq052268294>dir /b
code00.py
detector.cpp
(py35x64_test) e:WorkDevStackOverflowq052268294>cl /nologo /DDLL /EHsc detector.cpp /link /DLL /OUT:sim.dll
detector.cpp
Creating library sim.lib and object sim.exp
(py35x64_test) e:WorkDevStackOverflowq052268294>dir /b
code00.py
detector.cpp
detector.obj
sim.dll
sim.exp
sim.lib
(py35x64_test) e:WorkDevStackOverflowq052268294>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32
From C - [detector.cpp] (28) - [Detector::Detector]: &m_var: 0x0000020CE366E270
From C - [detector.cpp] (34) - [Detector::process]: &m_var: 0x0000020CE366E270
Done.
I wrote a dll library in c, compile with vs2017 64-bit, and try to load it with python3.6 64-bit. However the object’s member variable’s address got truncated to 32-bit.
Here’s my sim.c file, which is compiled to sim.dll:
class Detector {
public:
Detector();
void process(int* pin, int* pout, int n);
private:
int member_var;
};
Detector::Detector()
{
memset(&member_var, 0, sizeof(member_var));
myfile.open("addr_debug.txt");
myfile << "member_var init address: " << &member_var << endl;
}
void Detector::process(int* pin, int* pout, int n);
{
myfile << "member_var process address: " << &member_var << endl;
myfile.close();
}
#define DllExport __declspec( dllexport )
extern "C" {
DllExport Detector* Detector_new() { return new Detector(); }
DllExport void Detector_process(Detector* det, int* pin, int* pout, int n)
{
det->process(pin, pout, n);
}
}
Here’s my python script:
from ctypes import cdll
lib = cdll.LoadLibrary(r'sim.dll')
class Detector(object):
def __init__(self):
self.obj = lib.Detector_new()
def process(self,pin, pout, n):
lib.Detector_process(self.obj,pin, pout, n)
detector = Detector()
n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)
aptr = a.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
bptr = b.ctypes.data_as(ctypes.POINTER(ctypes.c_int))
detector.process(aptr, bptr, n)
Here’s the address of the member_var in addr_debug.txt:
member_var init address: 0000025259E123C4
member_var process address: 0000000059E123C4
So accessing it trigger memory access error:
OSError: exception: access violation reading 0000000059E123C4
Some attempts I tried to understand the issue:
- Define member_var as public instead of private, not help, address still truncated.
- Define member_var as global variable, then the address is ok. So I guess the member_var address truncation happens either when returning the object to python, or passing the object back to dll.
Always (CORRECTLY) specify argtypes and restype for functions defined in C, otherwise (C89 style) they will default to int (generally 32bit), generating !!! Undefined Behavior !!!. On 64bit, addresses (larger than 2 GiB) will be truncated (which is exactly what you’re experiencing). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati’s answer) for more details.
Also, when running into issues, don’t forget about [Python.Docs]: ctypes – A foreign function library for Python.
Below it’s an adapted version of your code.
detector.cpp:
#include <stdio.h>
#include <memory.h>
#include <fstream>
#define SIM_EXPORT __declspec(dllexport)
#define C_TAG "From C"
#define PRINT_MSG_3SPI(ARG0, ARG1, ARG2) printf("%s - [%s] (%d) - [%s]: %s: 0x%0p(%d)n", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1, ARG2)
using std::endl;
std::ofstream outFile;
class Detector {
public:
Detector();
void process(int *pIn, int *pOut, int n);
private:
int m_var;
};
Detector::Detector()
: m_var(25) {
outFile.open("addr_debug.txt");
outFile << "m_var init address: " << &m_var << endl;
PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}
void Detector::process(int *pIn, int *pOut, int n)
{
outFile << "m_var process address: " << &m_var << endl;
outFile.close();
PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}
#if defined(__cplusplus)
extern "C" {
#endif
SIM_EXPORT Detector* DetectorNew() { return new Detector(); }
SIM_EXPORT void DetectorProcess(Detector *pDet, int *pIn, int *pOut, int n)
{
pDet->process(pIn, pOut, n);
}
SIM_EXPORT void DetectorDelete(Detector *pDet) { delete pDet; }
#if defined(__cplusplus)
}
#endif
code00.py:
#!/usr/bin/env python
import ctypes as ct
import sys
import numpy as np
IntPtr = ct.POINTER(ct.c_int)
sim_dll = ct.CDLL("./sim.dll")
detector_new_func = sim_dll.DetectorNew
detector_new_func.argtypes = ()
detector_new_func.restype = ct.c_void_p
detector_process_func = sim_dll.DetectorProcess
detector_process_func.argtypes = (ct.c_void_p, IntPtr, IntPtr, ct.c_int)
detector_process_func.restype = None
detector_delete_func = sim_dll.DetectorDelete
detector_delete_func.argtypes = (ct.c_void_p,)
detector_delete_func.restype = None
class Detector():
def __init__(self):
self.obj = detector_new_func()
def process(self, pin, pout, n):
detector_process_func(self.obj, pin, pout, n)
def __del__(self):
detector_delete_func(self.obj)
def main(*argv):
detector = Detector()
n = 1024
a = np.arange(n, dtype=np.uint32)
b = np.zeros(n, dtype=np.int32)
aptr = a.ctypes.data_as(IntPtr)
bptr = b.ctypes.data_as(IntPtr)
detector.process(aptr, bptr, n)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}n".format(" ".join(elem.strip() for elem in sys.version.split("n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("nDone.")
sys.exit(rc)
Notes:
-
As I stated at the beginning, the problem was argtypes and restype not being specified (e.g. for DetectorNew: comment
detector_new_func.restype = ct.c_void_p, and you’ll run into the problem again) -
Code in the question is missing parts (#includes, imports, …), also there are some syntax errors, so it doesn’t compile, and therefore doesn’t follow [SO]: How to create a Minimal, Complete, and Verifiable example (mcve) guidelines. Please when make sure to have MCVE when asking
-
The object that you allocate (
new Detector()), must also be deallocated (otherwise, it will generate a memory leak), so I added a function (DetectorDelete – to do that), which is called from (Python) Detector‘s destructor -
Other (non critical) changes (identifiers renaming, a bit of refactoring, printing to stdout, …)
Output:
(py35x64_tes1) e:WorkDevStackOverflowq052268294>"c:Installx86MicrosoftVisual Studio Community2015vcvcvarsall.bat" x64 (py35x64_test) e:WorkDevStackOverflowq052268294>dir /b code00.py detector.cpp (py35x64_test) e:WorkDevStackOverflowq052268294>cl /nologo /DDLL /EHsc detector.cpp /link /DLL /OUT:sim.dll detector.cpp Creating library sim.lib and object sim.exp (py35x64_test) e:WorkDevStackOverflowq052268294>dir /b code00.py detector.cpp detector.obj sim.dll sim.exp sim.lib (py35x64_test) e:WorkDevStackOverflowq052268294>"e:WorkDevVEnvspy35x64_testScriptspython.exe" ./code.py Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32 From C - [detector.cpp] (28) - [Detector::Detector]: &m_var: 0x0000020CE366E270 From C - [detector.cpp] (34) - [Detector::process]: &m_var: 0x0000020CE366E270 Done.