Supplying NumPy site.cfg arguments to pip

I ended up putting together a script to automate this. Here it is, in case it can help someone else. I’ve tested it in Python 2.7, but it should work elsewhere without significant modifications.

from __future__ import unicode_literals

import io
import os.path
import re
import subprocess
import urllib2

# This downloads, builds, and installs NumPy against the MKL in the
# currently active virtualenv

file_name = 'numpy-1.6.2.tar.gz'
url = ('http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/'
       'numpy-1.6.2.tar.gz/download')

def main():

    # download NumPy and unpack it
    file_data = urllib2.urlopen(url).read()
    with io.open(file_name, 'wb') as fobj:
        fobj.write(file_data)
    subprocess.check_call('tar -xvf {0}'.format(file_name), shell=True)
    base_name = re.search(r'(.*).tar.gz$', file_name).group(1)
    os.chdir(base_name)

    # write out a site.cfg file in the build directory
    site_cfg = (
        '[mkl]n'
        'library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64n'
        'include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/includen'
        'mkl_libs = mkl_rtn'
        'lapack_libs =n')
    with io.open('site.cfg', 'wt', encoding='UTF-8') as fobj:
        fobj.write(site_cfg)

    # build and install NumPy
    subprocess.check_call('python setup.py build', shell=True)
    subprocess.check_call('python setup.py install', shell=True)


if __name__ == '__main__':
    main()

Your goal of installing NumPy to use Intel’s Math Kernel Library is now much easier since Intel created pips to install MKL + NumPy:

pip uninstall numpy -y  # if the standard numpy is present
pip install intel-numpy

as well as intel-scipy, intel-scikit-learn, pydaal, tbb4py, mkl_fft, mkl_random, and the lower level packages if you need just them. Again, you must first uninstall the standard packages if they’re already installed in your virtualenv.

NOTE:

If standard NumPy, SciPy and Scikit-Learn packages are already installed, the packages must be uninstalled before installing the Intel® variants of these packages(intel-numpy etc) to avoid any conflicts. As mentioned earlier, pydaal uses intel-numpy, hence it is important to first remove the standard Numpy library (if installed) and then install pydaal.

Alternatively, are there any environment variables that can be set instead of supplying the library paths in a site.cfg file?

NumPy 1.21 introduces environment variables for this purpose.

E.g.

NPY_BLAS_ORDER=MKL NPY_LAPACK_ORDER=MKL pip install numpy --no-binary numpy

to auto-detect the MKL library when installing NumPy from source code. If needed, you can set the environment variables NPY_BLAS_LIBS, NPY_CBLAS_LIBS, and NPY_LAPACK_LIBS to linker CLI options which put your chosen libraries on the linker path.

This is easier for a script to do than creating a ~/.numpy-site.cfg file,

[openblas]
libraries = openblas
library_dirs = /usr/local/opt/openblas/lib
include_dirs = /usr/local/opt/openblas/include
runtime_library_dirs = /usr/local/opt/openblas/lib

then running

pip install numpy --no-binary numpy

BTW, the file ~/.numpy-site.cfg also works when installing scipy from source code:

pip install scipy --no-binary scipy

NOTE: If you’re still using Python 2.7, install numpy then install scipy. Attempting to install them together will:

• invoke a SciPy easy_install installer that requests NumPy,
• load the latest NumPy installer (even if you specifically asked pip to install numpy==1.14.6 scipy==1.0.1 --no-binary numpy,scipy), then
• fail with RuntimeError: Python version >= 3.5 required because the latest NumPy does not support Python 2.7.