Using moviepy, scipy and numpy in amazon lambda

I was also following your first link and managed to import numpy and pandas in a Lambda function this way (on Windows):

1. Started a (free-tier) t2.micro EC2 instance with 64-bit Amazon Linux AMI 2015.09.1 and used Putty to SSH in.

2. Tried the same commands you used and the one recommended by the Amazon article:

   sudo yum -y update
   sudo yum -y upgrade
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install blas --enablerepo=epel
   sudo yum -y install lapack --enablerepo=epel
   sudo yum -y install Cython --enablerepo=epel
   sudo yum install python27-devel python27-pip gcc
   

3. Created the virtual environment:

   virtualenv ~/env
   source ~/env/bin/activate
   

4. Installed the packages:

   sudo ~/env/bin/pip2.7 install numpy
   sudo ~/env/bin/pip2.7 install pandas
   

5. Then, using WinSCP, I logged in and downloaded everything (except _markerlib, pip*, pkg_resources, setuptools* and easyinstall*) from /home/ec2-user/env/lib/python2.7/dist-packages, and everything from /home/ec2-user/env/lib64/python2.7/site-packages from the EC2 instance.

6. I put all these folders and files into one zip, along with the .py file containing the Lambda function.
   illustration of all files copied

7. Because this .zip is larger than 10 MB, I created an S3 bucket to store the file. I copied the link of the file from there and pasted at “Upload a .ZIP from Amazon S3” at the Lambda function.

8. The EC2 instance can be shut down, it’s not needed any more.

With this, I could import numpy and pandas. I’m not familiar with moviepy, but scipy might already be tricky as Lambda has a limit for unzipped deployment package size at 262 144 000 bytes. I’m afraid numpy and scipy together are already over that.

The posts here help me to find a way to statically compile NumPy with libraries files that can be included in the AWS Lambda Deployment package. This solution does not depend on LD_LIBRARY_PATH value as in @rouk1 solution.

The compiled NumPy library can be downloaded from https://github.com/vitolimandibhrata/aws-lambda-numpy

Here are the steps to custom compile NumPy

Instructions on compiling this package from scratch

Prepare a fresh AWS EC instance with AWS Linux.

Install compiler dependencies

sudo yum -y install python-devel
sudo yum -y install gcc-c++
sudo yum -y install gcc-gfortran
sudo yum -y install libgfortran

Install NumPy dependencies

sudo yum -y install blas
sudo yum -y install lapack
sudo yum -y install atlas-sse3-devel

Create /var/task/lib to contain the runtime libraries

mkdir -p /var/task/lib

/var/task is the root directory where your code will reside in AWS Lambda thus we need to statically link the required library files in a well known folder which in this case /var/task/lib

Copy the following library files to the /var/task/lib

cp /usr/lib64/atlas-sse3/liblapack.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libatlas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/libgfortran.so.3 /var/task/lib/.
cp /usr/lib64/libquadmath.so.0 /var/task/lib/.

Get the latest numpy source code from http://sourceforge.net/projects/numpy/files/NumPy/

Go to the numpy source code folder e.g numpy-1.10.4
Create a site.cfg file with the following entries

[atlas]
libraries=lapack,f77blas,cblas,atlas
search_static_first=true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

-lgfortran -lquadmath flags are required to statically link gfortran and quadmath libraries with files defined in runtime_library_dirs

Build NumPy

python setup.py build

Install NumPy

python setup.py install

Check whether the libraries are linked to the files in /var/task/lib

ldd $PYTHON_HOME/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so

You should see

linux-vdso.so.1 =>  (0x00007ffe0dd2d000)
liblapack.so.3 => /var/task/lib/liblapack.so.3 (0x00007ffad6be5000)
libptf77blas.so.3 => /var/task/lib/libptf77blas.so.3 (0x00007ffad69c7000)
libptcblas.so.3 => /var/task/lib/libptcblas.so.3 (0x00007ffad67a7000)
libatlas.so.3 => /var/task/lib/libatlas.so.3 (0x00007ffad6174000)
libf77blas.so.3 => /var/task/lib/libf77blas.so.3 (0x00007ffad5f56000)
libcblas.so.3 => /var/task/lib/libcblas.so.3 (0x00007ffad5d36000)
libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007ffad596d000)
libgfortran.so.3 => /var/task/lib/libgfortran.so.3 (0x00007ffad5654000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffad5352000)
libquadmath.so.0 => /var/task/lib/libquadmath.so.0 (0x00007ffad5117000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffad4f00000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffad4b3e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffad4922000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffad471d000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007ffad451a000)
/lib64/ld-linux-x86-64.so.2 (0x000055cfc3ab8000)

I like @Vito Limandibhrata’s answer but I think it’s not enough to build numpy with runtime_library_dirs in numpy==1.11.1. If anybody think site-cfg is ignored, do the following:

cp /usr/lib64/atlas-sse3/*.a /var/task/lib/

*.a files under atlas-sse3 are needed to build numpy. Also, you might need to run the following:

python setup.py config

to check numpy configuration. If it requires something more, you will see the following message:

atlas_threads_info:
Setting PTATLAS=ATLAS   libraries ptf77blas,ptcblas,atlas not found in /root/Envs/skl/lib
    libraries lapack_atlas not found in /root/Envs/skl/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64   
    libraries lapack_atlas not found in /usr/local/lib64
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib         
    libraries lapack_atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib64/atlas-sse3
<class 'numpy.distutils.system_info.atlas_threads_info'>
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
    libraries lapack not found in ['/var/task/lib']
Runtime library lapack was not found. Ignoring
    libraries f77blas not found in ['/var/task/lib']
Runtime library f77blas was not found. Ignoring
    libraries cblas not found in ['/var/task/lib']
Runtime library cblas was not found. Ignoring
    libraries atlas not found in ['/var/task/lib']
Runtime library atlas was not found. Ignoring
    FOUND:
        extra_link_args = ['-lgfortran -lquadmath']
        define_macros = [('NO_ATLAS_INFO', -1)]
        language = f77
        libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas', 'lapack', 'f77blas', 'cblas', 'atlas']
        library_dirs = ['/usr/lib64/atlas-sse3']
        include_dirs = ['/usr/include']

then site-cfg is going to be ignored.

Tip: If pip is used to build numpy with runtime_library_dirs, you would better create ~/.numpy-site.cfg and add the following:

[atlas]
libraries = lapack,f77blas,cblas,atlas
search_static_first = true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

then numpy recognizes .numpy-site.cfg file. It’s quite simple and easy way.

Another, very simple method that’s possible these days is to build using the awesome docker containers that LambCI made to mimic Lambda: https://github.com/lambci/docker-lambda

The lambci/lambda:build container resembles AWS Lambda with the addition of a mostly-complete build environment. To start a shell session in it:

docker run -v "$PWD":/var/task -it lambci/lambda:build bash

Inside the session:

export share=/var/task
easy_install pip
pip install -t $share numpy

Or, with virtualenv:

export share=/var/task
export PS1="[u@h:w]$ " # required by virtualenv
easy_install pip
pip install virtualenv
# ... make the venv, install numpy, and copy it to $share

Later on you can use the main lambci/lambda container to test your build.

As of 2017, NumPy and SciPy have wheels that work on Lambda (the packages include precompiled libgfortran and libopenblas).
As far as I know, MoviePy is a pure Python module, so basically you could do:

pip2 install -t lambda moviepy scipy

Then copy your handler into the lambda directory and zip it. Except, that you’ll most likely exceed the 50/250 MB size limits. There are a couple of things that can help:

• remove .pycs, docs, tests and other unnecessary parts;
• leave a single copy of common libraries of NumPy and SciPy;
• strip libraries of inessential pieces, such as debugging symbols;
• compress the archive using higher settings.

Here’s an example script that automates the above points.

I can confirm that the steps posted by @attila-tanyi work correctly under Amazon Linux. I would only add that there is no need to use an EC2, as there is an Amazon Linux docker container available from the default repository.

docker pull amazonlinux && docker run -it amazonlinux
# Follow @attila-tanyi steps
# Note - sudo is not necessary here

I use the Dockerfile embedded in my application to build and deploy to Lambda.

As of 2018, Steps to install external modules in Python3 on AWS EC2:

1. Launch EC2 on Amazon Linux AMI 201709.

2. ssh with putty using private and public key and become super user.

3. Install Python 3 and create virtual env, then make it default

   yum install python36 python36-virtualenv python36-pip
   
   virtualenv -p python3.6 /tmp/my_python_lib
   
   source /tmp/my_python_lib/bin/activate
   
   which python --to check which version s installed
   
   pip3 install  numpy
   

4. Copy files under site packages and dist packages into your local machhine using winscp.

   To find actual location use grep commands —

     grep -r dist-packages *. 
   

These packages could be inside both lib and lib64.

5. Site and dist packages will be under location:

   /tmp/my_python_lib/lib64/python3.6,
   /tmp/my_python_lib/lib/python3.6
   

6. Zip these packages along with your script file and upload to S3 which can be accessed in lambda.Instead of zipping the root folder you have to select all files and zip it or send to compressed folder.

Additional tips:

1. If you want to install all packages under one directory, you can use command:

    pip install --upgrade --target=/tmp/my_python_lib/lib/python3.6/dist-packages pandas
   

As of August 2018, probably the easiest way is to start a new AWS Cloud9 environment.
Then create a Lambda function inside the environment. Next run this into the Cloud9 command line:

    cd YourApplicationName
    /venv/bin/pip install scipy -t .
    /venv/bin/pip install numpy -t .
    /venv/bin/pip install moviepy -t .

Now I am able to import the modules in the lambda_handler function.

Nov 2018. Hi friends, this post is extremely helpful for me. However, the answers so far are not very automated. I wrote a Python script and tutorial here https://gist.github.com/steinwaywhw/6a6a25d594cc07146c60af943f74c16f to automate the creation of compiled Python packages using pip and virtualenv on EC2. Everything is Python (Boto3), no bash script, no Web console, no awscli.

There are one other change besides automation, which I think is an improvement. I downloaded the whole Python virtual environment from EC2 preserving its folder structures, instead of merging lib and lib64 packages all together. I never understand the intended meaning of merging those two folders. What if some packages override other packages, right? Plus, faking an official virtual environment is non-the-less a safer way to go than rolling your own.

For the downloaded virtual environment to work, the source code of the Lambda function adds some boilerplate code to update Python search path using sys.path. The intended sys.path of a Python virtual environment can be found by

• On your own machine, create a virtual environment and activate it.
• Run a Python script in this virtual environment and do print(sys.path) after import sys.
  You can start from there and modify as you see fit.

A snippet of the boilerplate code to add for a Lambda function in order to load numpy and other packages from my packaged virtual environment is pasted below. In my case, I loaded pandas_datareader which relies on numpy.

import os
import sys 

# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
workdir = os.getenv('LAMBDA_TASK_ROOT')
version = f'{sys.version_info[0]}.{sys.version_info[1]}'
additionals = [f'{workdir}/venv/lib64/python{version}/site-packages',
               f'{workdir}/venv/lib64/python{version}/lib-dynload',
               f'{workdir}/venv/lib64/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/site-packages']
sys.path = additionals + sys.path

import pandas_datareader as pdr

You can create zip file for lambda with scipy, numpy, moviepy and pandas on any OS.

https://pypi.org/project/scipy/#files

pypi has wheel files for different OS and you can download manylinux whl file and unzip. After that remove dist-info and pyc files and zip all.
The final zip file can be uploaded to S3 then and converted into lambda layer.

This tutorial is helpful to understand it deeply.