aws lambda – failed to find libmagic
Question:
I’m using in my lambda function the magic library to determine the file`s type.
I first deployed it to a local container to check that everything works.
My DockerFile :
FROM lambci/lambda:build-python3.8
WORKDIR /app
RUN mkdir -p .aws
COPY requirements.txt ./
COPY credentials /app/.aws/
RUN mv /app/.aws/ ~/.aws/
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir -r requirements.txt -t "/app/dependencies/"
WORKDIR /app/dependencies
RUN zip -r lambda.zip *
requirements.txt :
python-magic
libmagic
In my local container when I run tests on the lambda logic everything went ok and passed (including the part that uses the magic code..).
I created a zip that contains the lambda.py code and with the python dependencies (last 3 lines in the docker file).
When I upload the zip to aws and test the lambda I’m getting the following error :
{
"errorMessage": "Unable to import module 'lambda': failed to find libmagic. Check your installation",
"errorType": "Runtime.ImportModuleError"
}
As you can see, on my local container I’m using baseline image lambci/lambda:build-python3.8
that should be the same aws uses when the lambda is launching.
I tried also to add python-magic-bin==0.4.14 to the requirements.txt instead of the magic and libmagic but it didnt help either because it seems that this module is for windows.
Into the lambda.zip I put also the lambda.py which is the file that includes my lambda function :
import boto3
import urllib.parse
from io import BytesIO
import magic
def lambda_handler(event, context):
s3 = boto3.client("s3")
if event:
print("Event : ", event)
event_data = event["Records"][0]
file_name = urllib.parse.unquote_plus(event_data['s3']['object']['key'])
print("getting file: {}".format(file_name))
bucket_name = event_data['s3']['bucket']['name']
file_from_s3 = s3.get_object(Bucket=bucket_name, Key=file_name)
file_obj = BytesIO(file_from_s3['Body'].read())
print(magic.from_buffer(file_obj.read(2048)))
What am I doing wrong ?
Answers:
I didn’t find a way to solve this issue, therefore, I decided to use a different library called filetype .
Example how to use it :
file_type = filetype.guess(file_object)
if file_type is not None:
file_type = file_type.MIME
file_object.seek(0)
print("File type : {}".format(file_type))
if file_type == "application/gzip":
file_binary_content = gzip.GzipFile(filename=None, mode='rb', fileobj=file_object).read()
elif file_type == "application/zip":
zipfile = ZipFile(file_object)
file_binary_content = zipfile.read(zipfile.namelist()[0])
While using filetype
as suggested by other answers is much simpler, that library does not detect as many file types as magic
does.
You can make python-magic
work on aws lambda with python3.8
by doing the following:
- Add
libmagic.so.1
to a lib
folder at the root of the lambda package. This lib
folder will be automatically added to LD_LIBRARY_PATH
on aws lambda. This library can be found in /usr/lib64/libmagic.so.1
on an amazon linux ec2 instance for example.
- Create a magic file or take the one available on an amazon linux ec2 instance in
/usr/share/misc/magic
and add it to your lambda package.
- The Magic constructor from
python-magic
takes a magic_file
argument. Make this point to your magic file. You can then create the magic object with magic.Magic(magic_file='path_to_your_magic_file')
and then call any function from python-magic
you like on that object.
These steps are not necessary on the python3.7
runtime as those libraries are already present in aws lambda.
Maybe someone finds it helpful. This is how I make my lambda become friends with python-magic
. I used a local Ubuntu machine to create a zip-package and Python 3.7.
First, create a directory. The name doesn’t matter.
mkdir lambda-package
cd lambda-package
Then, install locally python-magic. I used pip3 for it.
pip3 install python-magic -t .
Now, in your lambda-package directory create or copy (if you already have some code in your lambda) a lambda_function.py
file. Be aware, Lambda expects such name. After this step you should have the following directory structure inside the lambda-package:
lambda-package
│ lambda_function.py
│ magic/
│ python_magic-0.4.24.dist-info/
Now, zip the contents of this directory. Remember to zip the contents only, not the folder itself. So, inside the lambda-package run:
zip -r lambda-package.zip .
Final step. Create a new Lambda function from scratch. Make sure to choose a proper runtime. When your function is created, click Upload from
-> choose Zip file
and upload lambda-package.zip file.
Now you will be able to import python-magic normally, like this:
import magic
That’s it!
p.s. The error failed to find libmagic
appears under Python 3.8 runtime. Python 3.7 works fine.
I’m using in my lambda function the magic library to determine the file`s type.
I first deployed it to a local container to check that everything works.
My DockerFile :
FROM lambci/lambda:build-python3.8
WORKDIR /app
RUN mkdir -p .aws
COPY requirements.txt ./
COPY credentials /app/.aws/
RUN mv /app/.aws/ ~/.aws/
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir -r requirements.txt -t "/app/dependencies/"
WORKDIR /app/dependencies
RUN zip -r lambda.zip *
requirements.txt :
python-magic
libmagic
In my local container when I run tests on the lambda logic everything went ok and passed (including the part that uses the magic code..).
I created a zip that contains the lambda.py code and with the python dependencies (last 3 lines in the docker file).
When I upload the zip to aws and test the lambda I’m getting the following error :
{
"errorMessage": "Unable to import module 'lambda': failed to find libmagic. Check your installation",
"errorType": "Runtime.ImportModuleError"
}
As you can see, on my local container I’m using baseline image lambci/lambda:build-python3.8
that should be the same aws uses when the lambda is launching.
I tried also to add python-magic-bin==0.4.14 to the requirements.txt instead of the magic and libmagic but it didnt help either because it seems that this module is for windows.
Into the lambda.zip I put also the lambda.py which is the file that includes my lambda function :
import boto3
import urllib.parse
from io import BytesIO
import magic
def lambda_handler(event, context):
s3 = boto3.client("s3")
if event:
print("Event : ", event)
event_data = event["Records"][0]
file_name = urllib.parse.unquote_plus(event_data['s3']['object']['key'])
print("getting file: {}".format(file_name))
bucket_name = event_data['s3']['bucket']['name']
file_from_s3 = s3.get_object(Bucket=bucket_name, Key=file_name)
file_obj = BytesIO(file_from_s3['Body'].read())
print(magic.from_buffer(file_obj.read(2048)))
What am I doing wrong ?
I didn’t find a way to solve this issue, therefore, I decided to use a different library called filetype .
Example how to use it :
file_type = filetype.guess(file_object)
if file_type is not None:
file_type = file_type.MIME
file_object.seek(0)
print("File type : {}".format(file_type))
if file_type == "application/gzip":
file_binary_content = gzip.GzipFile(filename=None, mode='rb', fileobj=file_object).read()
elif file_type == "application/zip":
zipfile = ZipFile(file_object)
file_binary_content = zipfile.read(zipfile.namelist()[0])
While using filetype
as suggested by other answers is much simpler, that library does not detect as many file types as magic
does.
You can make python-magic
work on aws lambda with python3.8
by doing the following:
- Add
libmagic.so.1
to alib
folder at the root of the lambda package. Thislib
folder will be automatically added toLD_LIBRARY_PATH
on aws lambda. This library can be found in/usr/lib64/libmagic.so.1
on an amazon linux ec2 instance for example. - Create a magic file or take the one available on an amazon linux ec2 instance in
/usr/share/misc/magic
and add it to your lambda package. - The Magic constructor from
python-magic
takes amagic_file
argument. Make this point to your magic file. You can then create the magic object withmagic.Magic(magic_file='path_to_your_magic_file')
and then call any function frompython-magic
you like on that object.
These steps are not necessary on the python3.7
runtime as those libraries are already present in aws lambda.
Maybe someone finds it helpful. This is how I make my lambda become friends with python-magic
. I used a local Ubuntu machine to create a zip-package and Python 3.7.
First, create a directory. The name doesn’t matter.
mkdir lambda-package
cd lambda-package
Then, install locally python-magic. I used pip3 for it.
pip3 install python-magic -t .
Now, in your lambda-package directory create or copy (if you already have some code in your lambda) a lambda_function.py
file. Be aware, Lambda expects such name. After this step you should have the following directory structure inside the lambda-package:
lambda-package
│ lambda_function.py
│ magic/
│ python_magic-0.4.24.dist-info/
Now, zip the contents of this directory. Remember to zip the contents only, not the folder itself. So, inside the lambda-package run:
zip -r lambda-package.zip .
Final step. Create a new Lambda function from scratch. Make sure to choose a proper runtime. When your function is created, click Upload from
-> choose Zip file
and upload lambda-package.zip file.
Now you will be able to import python-magic normally, like this:
import magic
That’s it!
p.s. The error failed to find libmagic
appears under Python 3.8 runtime. Python 3.7 works fine.