Ctypes: fast way to convert a return pointer to an array or Python list
Question:
I am using ctypes to pass an array pointer to a dll and return a pointer to an array of doubles that was created using malloc in the dll. On return to Python, I need a fast way to convert the pointer to an array or Python list.
I can use this list comp, but it’s slow because there are 320,000 data points:
list_of_results = [ret_ptr[i] for i in range(320000)]
Ideally I would create the array in Python and pass it to the dll, but I have to create it using malloc in the dll because this is a dynamic array where I don’t know beforehand how many data elements there will be (although the return pointer also returns the number of data elements, so I know how many there are on return to Python) — I use realloc to extend the array size dynamically in the dll; I can use realloc with a Python array, but a call to free() at the end is not guaranteed to work.
Here is the relevant Python code:
CallTest = hDLL.Main_Entry_fn
CallTest.argtypes = [ctypes.POINTER(ctypes.c_double), ctypes.c_int64]
CallTest.restype = ctypes.POINTER(ctypes.c_double)
ret_ptr = CallTest(DataArray, number_of_data_points)
list_of_results = [ret_ptr[i] for i in range(320000)]
So my question is: what is the fastest way to convert a pointer returned from a dll to a Python list or array? The method shown above is too slow.
Answers:
Slicing a ctypes array or pointer will automatically produce a list:
list_of_results = ret_ptr[:320000]
Depending on what you mean by “convert the pointer to an array” and what output types you can work with, you may be able to do better. For example, you can make a NumPy array backed by the buffer directly, with no data copying:
buffer_as_ctypes_array = ctypes.cast(ret_ptr, ctypes.POINTER(ctypes.c_double*320000))[0]
buffer_as_numpy_array = numpy.frombuffer(buffer_as_ctypes_array, numpy.float64)
This will of course break horribly if you deallocate the buffer while something still needs it.
I am using ctypes to pass an array pointer to a dll and return a pointer to an array of doubles that was created using malloc in the dll. On return to Python, I need a fast way to convert the pointer to an array or Python list.
I can use this list comp, but it’s slow because there are 320,000 data points:
list_of_results = [ret_ptr[i] for i in range(320000)]
Ideally I would create the array in Python and pass it to the dll, but I have to create it using malloc in the dll because this is a dynamic array where I don’t know beforehand how many data elements there will be (although the return pointer also returns the number of data elements, so I know how many there are on return to Python) — I use realloc to extend the array size dynamically in the dll; I can use realloc with a Python array, but a call to free() at the end is not guaranteed to work.
Here is the relevant Python code:
CallTest = hDLL.Main_Entry_fn
CallTest.argtypes = [ctypes.POINTER(ctypes.c_double), ctypes.c_int64]
CallTest.restype = ctypes.POINTER(ctypes.c_double)
ret_ptr = CallTest(DataArray, number_of_data_points)
list_of_results = [ret_ptr[i] for i in range(320000)]
So my question is: what is the fastest way to convert a pointer returned from a dll to a Python list or array? The method shown above is too slow.
Slicing a ctypes array or pointer will automatically produce a list:
list_of_results = ret_ptr[:320000]
Depending on what you mean by “convert the pointer to an array” and what output types you can work with, you may be able to do better. For example, you can make a NumPy array backed by the buffer directly, with no data copying:
buffer_as_ctypes_array = ctypes.cast(ret_ptr, ctypes.POINTER(ctypes.c_double*320000))[0]
buffer_as_numpy_array = numpy.frombuffer(buffer_as_ctypes_array, numpy.float64)
This will of course break horribly if you deallocate the buffer while something still needs it.