Cython: "fatal error: numpy/arrayobject.h: No such file or directory"
Question:
I’m trying to speed up the answer here using Cython. I try to compile the code (after doing the cygwinccompiler.py hack explained here), but get a fatal error: numpy/arrayobject.h: No such file or directory...compilation terminated error. Can anyone tell me if it’s a problem with my code, or some esoteric subtlety with Cython?
Below is my code.
import numpy as np
import scipy as sp
cimport numpy as np
cimport cython
cdef inline np.ndarray[np.int, ndim=1] fbincount(np.ndarray[np.int_t, ndim=1] x):
cdef int m = np.amax(x)+1
cdef int n = x.size
cdef unsigned int i
cdef np.ndarray[np.int_t, ndim=1] c = np.zeros(m, dtype=np.int)
for i in xrange(n):
c[<unsigned int>x[i]] += 1
return c
cdef packed struct Point:
np.float64_t f0, f1
@cython.boundscheck(False)
def sparsemaker(np.ndarray[np.float_t, ndim=2] X not None,
np.ndarray[np.float_t, ndim=2] Y not None,
np.ndarray[np.float_t, ndim=2] Z not None):
cdef np.ndarray[np.float64_t, ndim=1] counts, factor
cdef np.ndarray[np.int_t, ndim=1] row, col, repeats
cdef np.ndarray[Point] indices
cdef int x_, y_
_, row = np.unique(X, return_inverse=True); x_ = _.size
_, col = np.unique(Y, return_inverse=True); y_ = _.size
indices = np.rec.fromarrays([row,col])
_, repeats = np.unique(indices, return_inverse=True)
counts = 1. / fbincount(repeats)
Z.flat *= counts.take(repeats)
return sp.sparse.csr_matrix((Z.flat,(row,col)), shape=(x_, y_)).toarray()
Answers:
The error means that a numpy header file isn’t being found during compilation.
Try doing export CFLAGS=-I/usr/lib/python2.7/site-packages/numpy/core/include/, and then compiling. This is a problem with a few different packages. There’s a bug filed in ArchLinux for the same issue: https://bugs.archlinux.org/task/22326
In your setup.py, the Extension should have the argument include_dirs=[numpy.get_include()].
Also, you are missing np.import_array() in your code.
—
Example setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
setup(
ext_modules=[
Extension("my_module", ["my_module.c"],
include_dirs=[numpy.get_include()]),
],
)
# Or, if you use cythonize() to make the ext_modules list,
# include_dirs can be passed to setup()
setup(
ext_modules=cythonize("my_module.pyx"),
include_dirs=[numpy.get_include()]
)
For a one-file project like yours, another alternative is to use pyximport. You don’t need to create a setup.py … you don’t need to even open a command line if you use IPython … it’s all very convenient. In your case, try running these commands in IPython or in a normal Python script:
import numpy
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"],
"include_dirs":numpy.get_include()},
reload_support=True)
import my_pyx_module
print my_pyx_module.some_function(...)
...
You may need to edit the compiler of course. This makes import and reload work the same for .pyx files as they work for .py files.
Simple answer
A way simpler way is to add the path to your file distutils.cfg. It’s path behalf of Windows 7 is by default C:Python27Libdistutils. You just assert the following contents and it should work out:
[build_ext]
include_dirs= C:Python27Libsite-packagesnumpycoreinclude
Entire config file
To give you an example how the config file could look like, my entire file reads:
[build]
compiler = mingw32
[build_ext]
include_dirs= C:Python27Libsite-packagesnumpycoreinclude
compiler = mingw32
It should be able to do it within cythonize() function as mentioned here, but it doesn’t work beacuse there is a known issue
If you are too lazy to write setup files and figure out the path for include directories,
try cyper. It can compile your Cython code and set include_dirs for Numpy automatically.
Load your code into a string, then simply run cymodule = cyper.inline(code_string), then your function is available as cymodule.sparsemaker instantaneously. Something like this
code = open(your_pyx_file).read()
cymodule = cyper.inline(code)
cymodule.sparsemaker(...)
# do what you want with your function
You can install cyper via pip install cyper.
As per this answer, if you have installed numpy with pip on Linux, you will need to manually set a symbolic link to /usr/include/numpy
In my case the path is:
sudo ln -s /usr/local/lib/python3.8/dist-packages/numpy/core/include/numpy/ /usr/include/numpy
I hadn’t sudo privileges on the server I was running and export CFLAGS didn’t work with me. For sake of simplicity, I’ve installed Anaconda (
https://docs.anaconda.com/anaconda/install/) which creates links to all its installed packages, including Numpy. You can also install miniconda and work with environments to avoid using too much space.
I’m trying to speed up the answer here using Cython. I try to compile the code (after doing the cygwinccompiler.py hack explained here), but get a fatal error: numpy/arrayobject.h: No such file or directory...compilation terminated error. Can anyone tell me if it’s a problem with my code, or some esoteric subtlety with Cython?
Below is my code.
import numpy as np
import scipy as sp
cimport numpy as np
cimport cython
cdef inline np.ndarray[np.int, ndim=1] fbincount(np.ndarray[np.int_t, ndim=1] x):
cdef int m = np.amax(x)+1
cdef int n = x.size
cdef unsigned int i
cdef np.ndarray[np.int_t, ndim=1] c = np.zeros(m, dtype=np.int)
for i in xrange(n):
c[<unsigned int>x[i]] += 1
return c
cdef packed struct Point:
np.float64_t f0, f1
@cython.boundscheck(False)
def sparsemaker(np.ndarray[np.float_t, ndim=2] X not None,
np.ndarray[np.float_t, ndim=2] Y not None,
np.ndarray[np.float_t, ndim=2] Z not None):
cdef np.ndarray[np.float64_t, ndim=1] counts, factor
cdef np.ndarray[np.int_t, ndim=1] row, col, repeats
cdef np.ndarray[Point] indices
cdef int x_, y_
_, row = np.unique(X, return_inverse=True); x_ = _.size
_, col = np.unique(Y, return_inverse=True); y_ = _.size
indices = np.rec.fromarrays([row,col])
_, repeats = np.unique(indices, return_inverse=True)
counts = 1. / fbincount(repeats)
Z.flat *= counts.take(repeats)
return sp.sparse.csr_matrix((Z.flat,(row,col)), shape=(x_, y_)).toarray()
The error means that a numpy header file isn’t being found during compilation.
Try doing export CFLAGS=-I/usr/lib/python2.7/site-packages/numpy/core/include/, and then compiling. This is a problem with a few different packages. There’s a bug filed in ArchLinux for the same issue: https://bugs.archlinux.org/task/22326
In your setup.py, the Extension should have the argument include_dirs=[numpy.get_include()].
Also, you are missing np.import_array() in your code.
—
Example setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
setup(
ext_modules=[
Extension("my_module", ["my_module.c"],
include_dirs=[numpy.get_include()]),
],
)
# Or, if you use cythonize() to make the ext_modules list,
# include_dirs can be passed to setup()
setup(
ext_modules=cythonize("my_module.pyx"),
include_dirs=[numpy.get_include()]
)
For a one-file project like yours, another alternative is to use pyximport. You don’t need to create a setup.py … you don’t need to even open a command line if you use IPython … it’s all very convenient. In your case, try running these commands in IPython or in a normal Python script:
import numpy
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"],
"include_dirs":numpy.get_include()},
reload_support=True)
import my_pyx_module
print my_pyx_module.some_function(...)
...
You may need to edit the compiler of course. This makes import and reload work the same for .pyx files as they work for .py files.
Simple answer
A way simpler way is to add the path to your file distutils.cfg. It’s path behalf of Windows 7 is by default C:Python27Libdistutils. You just assert the following contents and it should work out:
[build_ext]
include_dirs= C:Python27Libsite-packagesnumpycoreinclude
Entire config file
To give you an example how the config file could look like, my entire file reads:
[build]
compiler = mingw32
[build_ext]
include_dirs= C:Python27Libsite-packagesnumpycoreinclude
compiler = mingw32
It should be able to do it within cythonize() function as mentioned here, but it doesn’t work beacuse there is a known issue
If you are too lazy to write setup files and figure out the path for include directories,
try cyper. It can compile your Cython code and set include_dirs for Numpy automatically.
Load your code into a string, then simply run cymodule = cyper.inline(code_string), then your function is available as cymodule.sparsemaker instantaneously. Something like this
code = open(your_pyx_file).read()
cymodule = cyper.inline(code)
cymodule.sparsemaker(...)
# do what you want with your function
You can install cyper via pip install cyper.
As per this answer, if you have installed numpy with pip on Linux, you will need to manually set a symbolic link to /usr/include/numpy
In my case the path is:
sudo ln -s /usr/local/lib/python3.8/dist-packages/numpy/core/include/numpy/ /usr/include/numpy
I hadn’t sudo privileges on the server I was running and export CFLAGS didn’t work with me. For sake of simplicity, I’ve installed Anaconda (
https://docs.anaconda.com/anaconda/install/) which creates links to all its installed packages, including Numpy. You can also install miniconda and work with environments to avoid using too much space.