Trouble installing scipy in virtualenv on a amazon ec2 linux micro instance

Question

I have successfully installed scipy in the default python compiler on an amazon ec2 micro instance (Ubuntu 13.04). However i am not able to install scipy in a virtualenv.

pip install scipy ends with this error

scipy/sparse/sparsetools/csr_wrap.cxx: In function ‘void init_csr()’:

scipy/sparse/sparsetools/csr_wrap.cxx:73303:21: warning: variable ‘md’ set but not used [-Wunused-but-set-variable]

c++: internal compiler error: Killed (program cc1plus)

Please submit a full bug report,

with preprocessed source if appropriate.

See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions.

----------------------------------------
Cleaning up...
Command /home/ubuntu/pnr/bin/python -c "import setuptools;__file__='/home/ubuntu/pnr/build/scipy/setup.py';exec(compile(open(__file__).read().replace('rn', 'n'), __file__, 'exec'))" install --record /tmp/pip-t8Drvd-record/install-record.txt --single-version-externally-managed --install-headers /home/ubuntu/pnr/include/site/python2.7 failed with error code -9 in /home/ubuntu/pnr/build/scipy

and

Traceback (most recent call last):
  File "/home/ubuntu/pnr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
  File "/home/ubuntu/pnr/local/lib/python2.7/site-packages/pip/__init__.py", line 148, in main
    return command.main(args[1:], options)
  File "/home/ubuntu/pnr/local/lib/python2.7/site-packages/pip/basecommand.py", line 169, in main
    text = 'n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 53: ordinal not in range(128)

Before anyone asks. pip freeze for default compiler returns

Cheetah==2.4.4
Landscape-Client==12.12
M2Crypto==0.21.1
PAM==0.4.2
Pillow==2.0.0
PyYAML==3.10
Twisted-Core==12.3.0
Twisted-Names==12.3.0
Twisted-Web==12.3.0
apt-xapian-index==0.45
argparse==1.2.1
boto==2.3.0
chardet==2.0.1
cloud-init==0.7.2
configobj==4.7.2
distribute==0.6.34
distro-info==0.10
euca2ools==2.1.1
numpy==1.7.1
oauth==1.0.1
paramiko==1.7.7.1
prettytable==0.6.1
pyOpenSSL==0.13
pycrypto==2.6
pycurl==7.19.0
pygobject==3.8.0
pyserial==2.6
python-apt==0.8.8ubuntu6
python-debian==0.1.21-nmu2ubuntu1
requests==1.1.0
scipy==0.11.0
six==1.2.0
ssh-import-id==3.14
urllib3==1.5
virtualenv==1.10.1
wsgiref==0.1.2
zope.interface==4.0.5

pip freeze command for virtualenv returns

Cython==0.19.2
Flask==0.10.1
Flask-Bootstrap==3.0.0.1
Flask-WTF==0.9.3
Jinja2==2.7.1
MarkupSafe==0.18
WTForms==1.0.5
Werkzeug==0.9.4
argparse==1.2.1
beautifulsoup4==4.3.2
itsdangerous==0.23
numpy==1.7.1
pymongo==2.6.2
requests==2.0.0
wsgiref==0.1.2

Asked By: Salil Panikkaveettil

||

Source

Answer 1

Yes, 512MB is not enough for compiling that C++ file.

Your best option is to build Scipy as a binary package (bdist, or eggs, or, more modern wheels) e.g. via python setupegg.py bdist_egg on a different machine with compatible environment. For instance, use a similar Linux version to the EC2 instance in a virtual machine.

In general, it’s good to remember that when pip installs packages, it compiles source files. If the package is not tiny, this is inefficient and it’s better to use binary packages. The wheel package format is supposed to play well together with pip.

Answered By: pv.

Answer 2

One solution is to temporarily enable swap on your micro instance. As described at this SO post, enable 1gb swap via:

sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
sudo /sbin/mkswap /var/swap.1
sudo /sbin/swapon /var/swap.1

Once swap is on, install scipy via pip:

sudo apt-get install -y libatlas-base-dev gfortran python-dev build-essential g++
sudo pip install numpy
sudo pip install scipy

Once scipy successfully installs, you can disable it via:

sudo swapoff /var/swap.1
sudo rm /var/swap.1

Answered By: Dolan Antenucci

Answer 3

for me the error was slightly different.

Blas (http://www.netlib.org/blas/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [blas]) or by setting

the BLAS environment variable.

----------------------------------------
Cleaning up...
Command /home/kdixit/pyvirt/bin/python -c "import setuptools;__file__='/home/kdixit/pyvirt/build/scipy/setup.py';exec(compile(open(__file__).read().replace('rn

Thus I had to install

sudo apt-get install libblas-dev

and then it worked.

Answered By: Kinjal Dixit

Answer 4

I was getting an out of virtual memory error during the scipy compilation on a t2.micro, I think numpy compiled just fine. Anyway, I guess that Dolan Antenucci‘s answer would fix my issue, but instead I went another route, a compromise, which does work.

I’m running an Ubuntu instance so keep that in mind.

sudo apt-get -y install python-scipy && echo -e "nok, installed python-scipy, continuing...n"

# add scipy to the venv
mkdir ~/venv_PROJECT/lib/python2.7/site-packages/scipy/
ln -s /usr/lib/python2.7/dist-packages/scipy/* ~/venv_PROJECT/lib/python2.7/site-packages/scipy/

# add numpy to the venv
mkdir ~/venv_PROJECT/lib/python2.7/site-packages/numpy/
ln -s /usr/lib/python2.7/dist-packages/numpy/* ~/venv_PROJECT/lib/python2.7/site-packages/numpy/

# add PIL to the venv
mkdir ~/venv_PROJECT/lib/python2.7/site-packages/PIL/
ln -s /usr/lib/python2.7/dist-packages/PIL/* ~/venv_PROJECT/lib/python2.7/site-packages/PIL/

The compromise is that you won’t be able to use diferent versions among different virtual environments and you’re tied to the version in the repo.

A word of warning:

>>> import scipy
>>> scipy.__version__
'0.13.3'

>>> import numpy
>>> numpy.__version__
'1.8.2'

>>> from PIL import Image
>>> Image.VERSION
'1.1.7'

Answered By: Daniel F

Answer 5

This worked for me:

pip --no-cache-dir install scipy

See:

note:

works for other service providers, hardware, VMs, and containers.
if RAM allocation size of 1GB
just calculate the diff between the cached directory memory usage and available ram

Answered By: jmunsch

Trouble installing scipy in virtualenv on a amazon ec2 linux micro instance

Question:

Answers: