Unable to import lxml etree on aws lambda

Question:

{
  "errorMessage": "Unable to import module 'lambda_function': 
        cannot import name   'etree' from 'lxml' (/var/task/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

Also tried https://gist.github.com/allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639:

{
  "errorMessage": "Unable to import module 'lambda_function': 
     cannot import name 'etree' from 'lxml' 
     (/var/task/lxml-4.3.4-py3.6-linux-x86_64.egg/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

I am running on Ubuntu 18.04 on my local machine, and have also tried using the “Amazon Linux” image on an ec2 instance to build the bundle.

I have also tried, while within the activated venv:

STATIC_DEPS=true pip3 install lxml --target ./package --upgrade --no-cache-dir

I have also tried copying the shared object files based on pulling which files were opened when running the script with strace:

#! /bin/bash

export Z=$(pwd)/ok-daily-lambda.zip
rm $Z
zip $Z lambda_function.py
zip $Z __init__.py

for dir in $(find venv_here/lib/python3.6/site-packages)
do
    if [ -d $dir ] ; then
        pushd $dir
        echo zip -r9 $Z $(pwd)
        zip -r9 $Z $(pwd)
        popd
    fi
done

export LIBD=$(pwd)/lib
mkdir -p $LIBD

cp "/home/jmunsch/.local/lib/python3.6/site-packages/.libs_cffi_backend/libffi-d78936b1.so.6.0.4" $LIBD
cp "/lib/x86_64-linux-gnu/libbz2.so.1.0" $LIBD
cp "/lib/x86_64-linux-gnu/libc.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libdl.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libexpat.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libgcc_s.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/liblzma.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libm.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_dns.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_files.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libpthread.so.0" $LIBD
cp "/lib/x86_64-linux-gnu/libresolv.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/librt.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libtinfo.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libudev.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libutil.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libuuid.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libz.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libffi.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/liblz4.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libmpdec.so.2" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libssl.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libzstd.so.1" $LIBD

zip -r $Z $LIBD

AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="xxx" AWS_DEFAULT_REGION="us-east-1" aws lambda update-function-code --function-name ok-today --zip-file fileb://ok-daily-lambda.zip

Here is the directory structure of the most recent zip file:

Related:

Asked By: jmunsch

||
Source

Answers:

There are modules that cannot be added directly into the site-packages directory to be recognised inside an AWS Lambda environment. When that happens, you have to get an Amazon Linux image from Docker repositories and make your own compiled environment in a container version that will run on AWS Lambda

For example, if you want to use Python 3.6 a good choice will be amazonlinux:2018.03 in case you want to install more packages e.g. pandas, numpy, scipy

    docker run -v $(pwd):/outputs -it amazonlinux:2018.03

Since Amazon Linux is based on Red Hat, you have to install via yum all dependencies when running docker and having already created your virtual environment

    yum update -y
    yum install -y 
      python36 
      python36-devel 
      python36-virtualenv 
      python34-setuptools 
      gcc 
      gcc-c++ 
      findutils 
      rsync 
      Cython 
      findutils 
      which 
      gzip 
      tar 
      man-pages 
      man 
      wget 
      make 
      zip

For lxml you will also need

    (lambda_docker) bash-4.2# yum install libxml2
    ...
    (lambda_docker) bash-4.2# yum install libxslt
    ...

You install the module as usual

    pip3.6 install lxml

You should see something like

    (lambda_docker) bash-4.2# pip3.6 install lxml
    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/2d/53/34a9f0c79c548e430148837892b6ae91adee571a0e8b6c17bd7ff9c2d12e/lxml-4.3.4-cp36-cp36m-manylinux1_x86_64.whl (5.7MB)
         |################################| 5.7MB 2.0MB/s 
    Installing collected packages: lxml

Then, create your function lambda_function.py and add it in the zipped package pushd-ed and popd-ed from you docker session

    from lxml import etree

    def lambda_handler(event, context):
        print(__name__)
        print(etree.LXML_VERSION)

After created

    zip -g site-packages.zip lambda_function.py

Before uploading, you can check that your zip file contains the lxml directories

    [jonathan@docker lambda_docker]$ unzip -l site-packages.zip 
    Archive:  site-packages.zip
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  06-29-2019 23:09   __pycache__/
          277  06-29-2019 23:09   __pycache__/easy_install.cpython-36.pyc
          126  06-29-2019 23:09   easy_install.py
          119  06-29-2019 23:29   lambda_function.py
            0  06-29-2019 23:21   lib/
            0  06-29-2019 23:39   lxml/
            0  06-29-2019 23:37   lxml-4.3.4.dist-info/
            4  06-29-2019 23:37   lxml-4.3.4.dist-info/INSTALLER
         2954  06-29-2019 23:37   lxml-4.3.4.dist-info/METADATA
        13384  06-29-2019 23:37   lxml-4.3.4.dist-info/RECORD
          109  06-29-2019 23:37   lxml-4.3.4.dist-info/WHEEL
            5  06-29-2019 23:37   lxml-4.3.4.dist-info/top_level.txt
         7668  06-29-2019 23:37   lxml/ElementInclude.py
          551  06-29-2019 23:37   lxml/__init__.py
            0  06-29-2019 23:37   lxml/__pycache__/
         3331  06-29-2019 23:37   lxml/__pycache__/ElementInclude.cpython-36.pyc
    ...

Now upload the zip and create an empty test {} in your lambda function

Result

    START RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64 Version: $LATEST
    lambda_function
    (4, 3, 4, 0)
    END RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64
    REPORT RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  Duration: 0.30 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 50 MB

If you prefer an image

enter image description here

Perfectly ready for AWS Lambda

Hope it helps (:

Answered By: jslipknot

I came across a similar problem and I figured out one quick workaround

Using pre-compiled build of lxml

Download https://github.com/shubh2502/aws-lambda-lxml

  1. Folder 3.6.4 and 3.8.0 are lxml versions

  2. Inside lxml there are two builds python27 and python36

  3. As per AWS Lambda python version choose either one of them

  4. Inside python27 and python36 there is lxml folder

  5. Zip code with lxml folder and make sure python has the same version

  6. In Case of AWS Lambda layer put lxml folder into this structure –

    python/lib/python3.6/site-packages

I spent lots of time in docker and building these stuff, this method was savior for me, I hope this will help you out

Answered By: Shubham Nigam

Reason:

The .so files compiled on a different OS. (In my case macOS.)

Solution:

The following article was very helpful:

How do I create a Lambda layer using a simulated Lambda environment with Docker?

All I had to do is:

cd /path_to_requirements_txt/

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.7" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"

It created python/lib/python3.7/site-packages/ with all the dependencies!

Answered By: Dipen Dadhaniya

If you using SERVERLESS for deployment:

Then, check the python version on your local environment as well as the lambda runtime version you are using

I was using Python 3.8 on my local and the lambda runtime version was using was Python 3.7
So I changed the lambda version to use Python 3.8

THAT WORKED FOR ME!!!

I hope this will help you out

Answered By: Bharat

I faced the same issue with lxml package on aws lambda, After much research and testing I reached at this solution:

  1. Import lxml like this:

    import lxml as l

  2. Then use etree as

    l.etree()

This worked for me on aws lambda

Answered By: Ishfaq Khawaja
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.