Cython Numpy warning about NPY_NO_DEPRECATED_API when using MemoryView
Question:
I am converting a Cython memoryview to a numpy array (to be able to use it in pure Python code):
from libc.stdlib cimport realloc
cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cpdef np.ndarray[DTYPE_t] compute(DTYPE_t[:,::1] data):
cdef unsigned int Nchannels = data.shape[0]
cdef unsigned int Ndata = data.shape[1]
cdef DTYPE_t* output = NULL
cdef DTYPE_t[::1] mv
output = <DTYPE_t*>realloc(output, Ndata*sizeof(output))
if not output:
raise MemoryError()
mv = <DTYPE_t[:Ndata]>output
mv[10:Ndata-10] = 0.0
# various calculations...
return np.asarray(mv, dtype=DTYPE, order='C')
It compiles, but the compiler gives the following warning:
/Users/vlad/anaconda/lib/python2.7/site-packages/numpy/core/include
/nump/npy_1_7_deprecated_api.h:15:2: warning:
"Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
I added the suggested directive in setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
filename = 'agents3.pyx'
agents_module = Extension(
'Agents',
sources = [filename],
define_macros = [('NPY_NO_DEPRECATED_API', 'NPY_1_7_API_VERSION')],
include_dirs = [numpy.get_include()],
)
setup (name = 'Agents',
ext_modules = cythonize(agents_module)
)
Now it wouldn’t compile, it says:
Vlads-MacBook-Pro:program vlad$ python setup.py build_ext --inplace
Compiling agents3.pyx because it changed.
Cythonizing agents3.pyx
running build_ext
building 'Agents' extension
gcc -fno-strict-aliasing -I/Users/vlad/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/vlad/anaconda/lib/python2.7/site-packages/numpy/core/include -I/Users/vlad/anaconda/include/python2.7 -c agents3.c -o build/temp.macosx-10.5-x86_64-2.7/agents3.o
agents3.c:2273:52: error: use of undeclared identifier 'NPY_C_CONTIGUOUS'
__pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_C_CONTIGUOUS) != 0)) != 0);
^
agents3.c:2311:52: error: use of undeclared identifier 'NPY_F_CONTIGUOUS'
__pyx_t_1 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_F_CONTIGUOUS) != 0)) != 0);
^
agents3.c:2474:42: error: no member named 'descr' in 'struct tagPyArrayObject'
__pyx_t_4 = ((PyObject *)__pyx_v_self->descr);
~~~~~~~~~~~~ ^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:34: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:772:24: note: expanded from macro 'Py_DECREF'
--((PyObject*)(op))->ob_refcnt != 0)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:775:34: note: expanded from macro 'Py_DECREF'
_Py_Dealloc((PyObject *)(op));
^
/Users/vlad/anaconda/include/python2.7/object.h:762:15: note: expanded from macro '_Py_Dealloc'
(*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))
^
/Users/vlad/anaconda/include/python2.7/object.h:115:47: note: expanded from macro 'Py_TYPE'
#define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:775:34: note: expanded from macro 'Py_DECREF'
_Py_Dealloc((PyObject *)(op));
^
/Users/vlad/anaconda/include/python2.7/object.h:762:45: note: expanded from macro '_Py_Dealloc'
(*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))
^
agents3.c:4035:16: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_v_arr->base = __pyx_v_baseptr;
~~~~~~~~~~~ ^
agents3.c:4070:30: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_t_1 = ((__pyx_v_arr->base == NULL) != 0);
~~~~~~~~~~~ ^
agents3.c:4093:44: error: no member named 'base' in 'struct tagPyArrayObject'
__Pyx_INCREF(((PyObject *)__pyx_v_arr->base));
~~~~~~~~~~~ ^
agents3.c:1065:37: note: expanded from macro '__Pyx_INCREF'
#define __Pyx_INCREF(r) Py_INCREF(r)
^
/Users/vlad/anaconda/include/python2.7/object.h:767:18: note: expanded from macro 'Py_INCREF'
((PyObject*)(op))->ob_refcnt++)
^
agents3.c:4094:41: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_r = ((PyObject *)__pyx_v_arr->base);
~~~~~~~~~~~ ^
11 errors generated.
error: command 'gcc' failed with exit status 1
Vlads-MacBook-Pro:program vlad$
What should I do? Is it OK to leave the deprecated API call as it is? It tries to acces the base field — but I am not doing it, it’s Cython’s fault. I simply converted a memoryview to a numpy array. Is there another, cleaner/safer way of doing it?
Answers:
I also get the same warnings, and I’d say it’s normal.
With the numpy C API you need to put a line in front of the C script if you don’t like this warning, but all it does is tell the compiler to ignore the “deprecated” message – it seems to work the same either way.
I’m guessing the Cython compiler doesn’t put this line of code when it generates the C code, and I don’t think that’s important.
Just for a further reference, cython online docs says this is because Cython is using a deprecated Numpy API, and for the time being, it’s just a warning that we can ignore.
Assuming one wishes to hide the deprecation warning, the following compiler flag can be implemented with clang: extra_compile_args=['-Wno-#warnings'].
For gcc, extra_compile_args=['-Wno-cpp'] achieves the same.
Of course this also hides other preprocessor directive warnings.
In Cython 3.0, you can get rid of this warning by defining the C macro NPY_NO_DEPRECATED_API as NPY_1_7_API_VERSION in your build:
# distutils: define_macros=NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION
…
With older Cython releases, setting this macro will fail the C compilation, because Cython generates code that uses this deprecated C-API. However, the warning has no negative effects even in recent NumPy versions including 1.18.x. You can ignore it until you (or your library’s users) switch to a newer NumPy version that removes this long deprecated API, in which case you also need to use Cython 3.0 or later. Thus, the earlier you switch to Cython 3.0, the better for your users.
As of February 2023, Cython 3.0 has not been released, so that this advice to upgrade to Cython 3.0 to remove the warning didn’t age well.
You can ignore it advice isn’t the best practice either because it causes developer warning fatigue, apathy and ignorance, which leads to bugs.
In earlier Cython versions this warning can be disabled by defining C preprocessor macro NPY_NO_DEPRECATED_API=1, e.g.:
# distutils: define_macros=NPY_NO_DEPRECATED_API=1
I am converting a Cython memoryview to a numpy array (to be able to use it in pure Python code):
from libc.stdlib cimport realloc
cimport numpy as np
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
cpdef np.ndarray[DTYPE_t] compute(DTYPE_t[:,::1] data):
cdef unsigned int Nchannels = data.shape[0]
cdef unsigned int Ndata = data.shape[1]
cdef DTYPE_t* output = NULL
cdef DTYPE_t[::1] mv
output = <DTYPE_t*>realloc(output, Ndata*sizeof(output))
if not output:
raise MemoryError()
mv = <DTYPE_t[:Ndata]>output
mv[10:Ndata-10] = 0.0
# various calculations...
return np.asarray(mv, dtype=DTYPE, order='C')
It compiles, but the compiler gives the following warning:
/Users/vlad/anaconda/lib/python2.7/site-packages/numpy/core/include
/nump/npy_1_7_deprecated_api.h:15:2: warning:
"Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
I added the suggested directive in setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
filename = 'agents3.pyx'
agents_module = Extension(
'Agents',
sources = [filename],
define_macros = [('NPY_NO_DEPRECATED_API', 'NPY_1_7_API_VERSION')],
include_dirs = [numpy.get_include()],
)
setup (name = 'Agents',
ext_modules = cythonize(agents_module)
)
Now it wouldn’t compile, it says:
Vlads-MacBook-Pro:program vlad$ python setup.py build_ext --inplace
Compiling agents3.pyx because it changed.
Cythonizing agents3.pyx
running build_ext
building 'Agents' extension
gcc -fno-strict-aliasing -I/Users/vlad/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/vlad/anaconda/lib/python2.7/site-packages/numpy/core/include -I/Users/vlad/anaconda/include/python2.7 -c agents3.c -o build/temp.macosx-10.5-x86_64-2.7/agents3.o
agents3.c:2273:52: error: use of undeclared identifier 'NPY_C_CONTIGUOUS'
__pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_C_CONTIGUOUS) != 0)) != 0);
^
agents3.c:2311:52: error: use of undeclared identifier 'NPY_F_CONTIGUOUS'
__pyx_t_1 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_F_CONTIGUOUS) != 0)) != 0);
^
agents3.c:2474:42: error: no member named 'descr' in 'struct tagPyArrayObject'
__pyx_t_4 = ((PyObject *)__pyx_v_self->descr);
~~~~~~~~~~~~ ^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:34: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:772:24: note: expanded from macro 'Py_DECREF'
--((PyObject*)(op))->ob_refcnt != 0)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:775:34: note: expanded from macro 'Py_DECREF'
_Py_Dealloc((PyObject *)(op));
^
/Users/vlad/anaconda/include/python2.7/object.h:762:15: note: expanded from macro '_Py_Dealloc'
(*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))
^
/Users/vlad/anaconda/include/python2.7/object.h:115:47: note: expanded from macro 'Py_TYPE'
#define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)
^
agents3.c:4026:27: error: no member named 'base' in 'struct tagPyArrayObject'
Py_XDECREF(__pyx_v_arr->base);
~~~~~~~~~~~ ^
/Users/vlad/anaconda/include/python2.7/object.h:823:64: note: expanded from macro 'Py_XDECREF'
#define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
^
/Users/vlad/anaconda/include/python2.7/object.h:775:34: note: expanded from macro 'Py_DECREF'
_Py_Dealloc((PyObject *)(op));
^
/Users/vlad/anaconda/include/python2.7/object.h:762:45: note: expanded from macro '_Py_Dealloc'
(*Py_TYPE(op)->tp_dealloc)((PyObject *)(op)))
^
agents3.c:4035:16: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_v_arr->base = __pyx_v_baseptr;
~~~~~~~~~~~ ^
agents3.c:4070:30: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_t_1 = ((__pyx_v_arr->base == NULL) != 0);
~~~~~~~~~~~ ^
agents3.c:4093:44: error: no member named 'base' in 'struct tagPyArrayObject'
__Pyx_INCREF(((PyObject *)__pyx_v_arr->base));
~~~~~~~~~~~ ^
agents3.c:1065:37: note: expanded from macro '__Pyx_INCREF'
#define __Pyx_INCREF(r) Py_INCREF(r)
^
/Users/vlad/anaconda/include/python2.7/object.h:767:18: note: expanded from macro 'Py_INCREF'
((PyObject*)(op))->ob_refcnt++)
^
agents3.c:4094:41: error: no member named 'base' in 'struct tagPyArrayObject'
__pyx_r = ((PyObject *)__pyx_v_arr->base);
~~~~~~~~~~~ ^
11 errors generated.
error: command 'gcc' failed with exit status 1
Vlads-MacBook-Pro:program vlad$
What should I do? Is it OK to leave the deprecated API call as it is? It tries to acces the base field — but I am not doing it, it’s Cython’s fault. I simply converted a memoryview to a numpy array. Is there another, cleaner/safer way of doing it?
I also get the same warnings, and I’d say it’s normal.
With the numpy C API you need to put a line in front of the C script if you don’t like this warning, but all it does is tell the compiler to ignore the “deprecated” message – it seems to work the same either way.
I’m guessing the Cython compiler doesn’t put this line of code when it generates the C code, and I don’t think that’s important.
Just for a further reference, cython online docs says this is because Cython is using a deprecated Numpy API, and for the time being, it’s just a warning that we can ignore.
Assuming one wishes to hide the deprecation warning, the following compiler flag can be implemented with clang: extra_compile_args=['-Wno-#warnings'].
For gcc, extra_compile_args=['-Wno-cpp'] achieves the same.
Of course this also hides other preprocessor directive warnings.
In Cython 3.0, you can get rid of this warning by defining the C macro
NPY_NO_DEPRECATED_APIasNPY_1_7_API_VERSIONin your build:# distutils: define_macros=NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION…
With older Cython releases, setting this macro will fail the C compilation, because Cython generates code that uses this deprecated C-API. However, the warning has no negative effects even in recent NumPy versions including 1.18.x. You can ignore it until you (or your library’s users) switch to a newer NumPy version that removes this long deprecated API, in which case you also need to use Cython 3.0 or later. Thus, the earlier you switch to Cython 3.0, the better for your users.
As of February 2023, Cython 3.0 has not been released, so that this advice to upgrade to Cython 3.0 to remove the warning didn’t age well.
You can ignore it advice isn’t the best practice either because it causes developer warning fatigue, apathy and ignorance, which leads to bugs.
In earlier Cython versions this warning can be disabled by defining C preprocessor macro NPY_NO_DEPRECATED_API=1, e.g.:
# distutils: define_macros=NPY_NO_DEPRECATED_API=1