How to use ctypes.util.find_library to import .so libraries in AWS lambda (python)?

Question

What I’m trying

A python package I’m using (OCRMYPDF) on Lambda needs the leptonica library liblept.so.5. On isolating the import code I found the issue is with find_library(‘lept’). Printing the result returns None.

from ctypes.util import find_library
def lambda_handler(event, context):
    liblept=find_library('lept')
    print("liblept:%s"%liblept)

The python package I’m using needs many native compiled dependencies. I’m trying to import these using lambda layers.

layer structure

/opt/
  /opt/bin/
  /opt/lib/
    /opt/lib/liblept.so.5
  /opt/tesseract

I’m able to access the file using CDLL (code below). But I don’t want to rewrite the package and replace every find_library() with CDLL. Is it possible to set the import directory for find_library?

liblept=CDLL("/opt/lib/liblept.so.5") # found
print("liblept:%s"%liblept)

My layer code works

I used a docker image to build layer. The files in /opt/bin that depend on leptonica are working (tesseract runs properly, tested OCR as well).

logging.info(os.system("tesseract --version"))

output

START RequestId: d826d36c-4ce9-4b67-b501-8c9042edcf80 Version: $LATEST
tesseract 4.1.0
 leptonica-1.78.0
  libgif 5.1.4 : libjpeg 6b (libjpeg-turbo 1.2.90) : libpng 1.2.49 : libtiff 4.0.3 : zlib 1.2.8 : libwebp 0.3.0
 Found AVX
 Found SSE
END RequestId: d826d36c-4ce9-4b67-b501-8c9042edcf80

Asked By: secretshardul

||

Source

Answer 1

Tested on Python:3.7 AWS lambda environment:

You need to add liblept.so (just rename liblept.so.5) to the /lib folder of your lambda package or in /opt/lib of your layer. The file must be called liblept.so because find_library only looks for ".so" files, not ".so.5" files:

From the python doc: https://docs.python.org/3/library/ctypes.html

ctypes.util.find_library(name)
Try to find a library and return a pathname. name is the library name without any prefix like lib, suffix like .so, .dylib or version number (this is the form used for the posix linker option -l). If no library can be found, returns None.

When only adding "liblept.so", the linker complains that it cannot find "liblept.so.5", so I also added "liblept.so.5" to the lib folder.

Maybe someone else can pitch in and find a solution that does not use a duplicate file.

AWS lambda will automatically make any file in /opt/lib or /lib available via LD_LIBRARY_PATH.

On Python 3.8 you may also need to include ld and objdump, as per this thread: https://forums.aws.amazon.com/thread.jspa?threadID=313506, altough I have not tested that.

Answered By: mananony

Answer 2

For Amazon Linux 2 based runtimes (Python 3.8+), the utilities required to perform ctypes.util.find_library such as ldconfig, ld, and gcc have all been removed.

To correct this, I made a Lambda Container Image to take the base AWS runtime image and add gcc which will locate the desired .so file.

Dockerfile example

FROM public.ecr.aws/lambda/python:3.9-arm64

RUN yum install -y 
    gcc 
    && yum clean all

COPY requirements.txt ./

RUN python3.9 -m pip install -r requirements.txt

COPY *.py ./

Answered By: roktechie

How to use ctypes.util.find_library to import .so libraries in AWS lambda (python)?

Question:

What I’m trying

layer structure

My layer code works

Answers: