Storing unsafe C derivative of temporary Python reference error in Cython

Question

I was writing code to store a (potentially) very large integer value into an array of chars referenced by a pointer. My code looks like this:

cdef class Variable:

    cdef unsigned int Length
    cdef char * Array

    def __cinit__(self, var, length):
        self.Length = length
        self.Array = <char *>malloc(self.Length * sizeof(char))    # Error
        for i in range(self.Length):
            self.Array[i] = <char>(var >> (8 * i))

    def __dealloc__(self):
        self.Array = NULL

When I tried compiling the code, I got the error, “Storing unsafe C derivative of temporary Python reference” at the commented line. My question is this: which temporary Python reference am I deriving in C and storing, and how do I fix it?

Asked By: Woody1193

||

Source

Answer 1

The problem is that under the hood a temporary variable is being created to hold the array before the assignment to self.Array and it will not be valid once the method exits.

Note that the documentation advises:

the C-API functions for allocating memory on the Python heap are generally preferred over the low-level C functions above as the memory they provide is actually accounted for in Python’s internal memory management system. They also have special optimisations for smaller memory blocks, which speeds up their allocation by avoiding costly operating system calls.

Accordingly you can write as below, which seems to handle this use case as intended:

from cpython.mem cimport PyMem_Malloc, PyMem_Realloc, PyMem_Free

cdef class Variable:

    cdef unsigned int Length
    cdef char * Array

    def __cinit__(self, var,size_t length):
        self.Length = length
        self.Array = <char *>PyMem_Malloc(length * sizeof(char))
        #as in docs, a good practice
        if not self.Array:
            raise MemoryError()

        for i in range(self.Length):
            self.Array[i] = <char>(var >> (8 * i))

    def __dealloc__(self):
        PyMem_Free(self.Array)

Answered By: rll

Answer 2

@rll’s answer does a pretty good job of cleaning up the code and “doing everything properly” (most importantly the deallocation of memory which was missing from __dealloc__ in the question!).

The actual issue causing the error is that you haven’t cimported malloc. Because of this Cython assumes that malloc is a Python function, returning a Python object which you want to cast to a char*. At the top of your file, add

from libc.stdlib cimport malloc, free

and it’ll work. Or alternatively use PyMem_Malloc (cimporting it) as @rll does and that will work fine too.

Answered By: DavidW

Storing unsafe C derivative of temporary Python reference error in Cython

Question:

Answers: