AWS Lambda: Unable to import module 'python_handler': No module named '_cffi_backend'
Question:
I’m creating a AWS Lambda function when I need to read an info on an API, create a CSV file, and upload them on a SFTP server.
I’ve installed paramiko on my venv, using Ubuntu on Windows, and the cffi module comes like a dependency, but when the code runs, I receive this error:
{
"errorMessage": "Unable to import module 'python_handler': No module named '_cffi_backend'",
"errorType": "Runtime.ImportModuleError"
}
Follow my code:
import paramiko
import requests
from random import randrange
from datetime import datetime
from datetime import timedelta
from requests.auth import HTTPBasicAuth
def lambda_handler(event, context):
# Lambda function
addZero = lambda x : '0' + str(x) if x < 10 else str(x)
# Actual datetime
ac_yr = datetime.utcnow().year
ac_mo = datetime.utcnow().month
ac_da = datetime.utcnow().day
ac_hh = datetime.utcnow().hour
ac_mi = datetime.utcnow().minute
ac_se = datetime.utcnow().second
# 24 hours ago
ag_yr = (datetime.utcnow() - timedelta(hours=24)).year
ag_mo = (datetime.utcnow() - timedelta(hours=24)).month
ag_da = (datetime.utcnow() - timedelta(hours=24)).day
ag_hh = (datetime.utcnow() - timedelta(hours=24)).hour
ag_mi = (datetime.utcnow() - timedelta(hours=24)).minute
ag_se = (datetime.utcnow() - timedelta(hours=24)).second
# API Infos
api_key = 'XYZ'
page_id = 'XYZ'
# Call API
param = {
'sort_order': 'asc',
'from': '{}-{}-{}T{}:{}:{}.000Z'.format(ag_yr, addZero(ag_mo), addZero(ag_da), addZero(ag_hh), addZero(ag_mi), addZero(ag_se)),
'to': '{}-{}-{}T{}:{}:{}.000Z'.format(ac_yr, addZero(ac_mo), addZero(ac_da), addZero(ac_hh), addZero(ac_mi), addZero(ac_se))
}
r = requests.get('https://api.unbounce.com/pages/{}/leads'.format(page_id), auth=HTTPBasicAuth(api_key, 'pass'), params=param)
# If connection it's ok
if r.status_code == 200:
# If has any result
if len(r.json()['leads']) > 0:
cont = ''
for lead in r.json()['leads']:
cont += lead['form_data']['cpf'][0] + ','
cont += lead['form_data']['nome_completo'][0] + ','
cont += lead['form_data']['email'][0]
else:
return 'Não há resultados no momento'
else:
return 'Falha na conexão'
# Write a CSV file
f = open('my_csv.csv','w')
f.write('PAC_ID, PAC_NAME, PAC_EMAILn')
f.write(cont)
f.close()
transport = paramiko.Transport(('host-info', 22))
transport.connect(None, 'user-info', 'password-info', None)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.chdir('Import')
sftpclient.put('my_csv.csv')
return 'OK'
Any idea how I can solve this?
Answers:
This looks like a common problem with many binary packages.
I don’t have a generic solution yet, but what you really have to do is to move (symlink) the binary file from where it is really installed to the root of your package.
I usually solve this with post-install scripts for my Lambdas. The ones using paramiko run something like:
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so MY_LAMBDA/
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/.libs_cffi_backend/libffi-XXXXXXXX.so.6.0.4 MY_LAMBDA/
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/nacl MY_LAMBDA/
You will have to identify the exact name of the libffi... file from your virtualenv. It might vary from the machine you compile the library on.
Working on Ubuntu, I haven’t the success with pysftp or paramiko to Lambda.
So I created an EC2 instance (and after an VirtualBox with Amazon Linux 2) on my desktop and develop the same code, with the same libraries. And… that’s works…
may be late, hopefully, will help someone in future. this answer is specific to “_cffi_backend…..”. One other thing that may help is to know how you installed python. if did using brew, you will need to create a virtual env using python. see here:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
i did this in a linux environment without downloading the wheel files that are compatible with various linux distros and it worked
The simplest solution is to use the lambci/lambda container as described here. I wrote about it in some detail here.
In a nutshell, some Python packages (e.g cffi) have OS level dependencies. Because the OS you’re packaging the code on is different to the one Lambda uses, the script fails to run.
You can also get the error No module named '_cffi_backend' when the runtime version of your AWS Lambda function is different than the python version used to create your Lambda Layer.
I received this error when I set the runtime of my Lambda Function to Python 3.10, but I installed the dependencies for my Lambda Layer in an environment that was running Python 3.8.
Using serverless cli with serverless-python-requirements plugin, if you change the AWS Lambda python runtime version and publish, the plugin will use the same pip download cache which was using the old python version. (This cache is outside your git repo, so git clean doesn’t help either.)
That would resulting in having _cffi_backend.cpython-310-x86_64-linux-gnu.so in your lambda, even though you expect to use cpython-311.
The fix is to run serverless requirements cleanCache to delete the pip download cache.
Ran into the same issue while creating a layer for snowflake-python-connector for arm64 using amazonlinux docker container.
About the snowflake-python-connector issue in case it helps someone – packaging the layer using docker on x86 arch gives the following error
{ "errorMessage": "Unable to import module 'lambda_function': /opt/python/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError", "requestId": "07fc4b23-21c2-44e8-a6cd-7b918b84b9f9", "stackTrace": [] }
And using arm64 throws the following error
File "/usr/lib/python2.6/site-packages/cffi/api.py", line 56, in __init__
import _cffi_backend as backend
ImportError: No module named _cffi_backend
Creating the layer as mentioned in the link shared by OzNetNerd under the accepted solution comments worked for me. Posting the link here – https://repost.aws/knowledge-center/lambda-layer-simulated-docker
I’m creating a AWS Lambda function when I need to read an info on an API, create a CSV file, and upload them on a SFTP server.
I’ve installed paramiko on my venv, using Ubuntu on Windows, and the cffi module comes like a dependency, but when the code runs, I receive this error:
{
"errorMessage": "Unable to import module 'python_handler': No module named '_cffi_backend'",
"errorType": "Runtime.ImportModuleError"
}
Follow my code:
import paramiko
import requests
from random import randrange
from datetime import datetime
from datetime import timedelta
from requests.auth import HTTPBasicAuth
def lambda_handler(event, context):
# Lambda function
addZero = lambda x : '0' + str(x) if x < 10 else str(x)
# Actual datetime
ac_yr = datetime.utcnow().year
ac_mo = datetime.utcnow().month
ac_da = datetime.utcnow().day
ac_hh = datetime.utcnow().hour
ac_mi = datetime.utcnow().minute
ac_se = datetime.utcnow().second
# 24 hours ago
ag_yr = (datetime.utcnow() - timedelta(hours=24)).year
ag_mo = (datetime.utcnow() - timedelta(hours=24)).month
ag_da = (datetime.utcnow() - timedelta(hours=24)).day
ag_hh = (datetime.utcnow() - timedelta(hours=24)).hour
ag_mi = (datetime.utcnow() - timedelta(hours=24)).minute
ag_se = (datetime.utcnow() - timedelta(hours=24)).second
# API Infos
api_key = 'XYZ'
page_id = 'XYZ'
# Call API
param = {
'sort_order': 'asc',
'from': '{}-{}-{}T{}:{}:{}.000Z'.format(ag_yr, addZero(ag_mo), addZero(ag_da), addZero(ag_hh), addZero(ag_mi), addZero(ag_se)),
'to': '{}-{}-{}T{}:{}:{}.000Z'.format(ac_yr, addZero(ac_mo), addZero(ac_da), addZero(ac_hh), addZero(ac_mi), addZero(ac_se))
}
r = requests.get('https://api.unbounce.com/pages/{}/leads'.format(page_id), auth=HTTPBasicAuth(api_key, 'pass'), params=param)
# If connection it's ok
if r.status_code == 200:
# If has any result
if len(r.json()['leads']) > 0:
cont = ''
for lead in r.json()['leads']:
cont += lead['form_data']['cpf'][0] + ','
cont += lead['form_data']['nome_completo'][0] + ','
cont += lead['form_data']['email'][0]
else:
return 'Não há resultados no momento'
else:
return 'Falha na conexão'
# Write a CSV file
f = open('my_csv.csv','w')
f.write('PAC_ID, PAC_NAME, PAC_EMAILn')
f.write(cont)
f.close()
transport = paramiko.Transport(('host-info', 22))
transport.connect(None, 'user-info', 'password-info', None)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.chdir('Import')
sftpclient.put('my_csv.csv')
return 'OK'
Any idea how I can solve this?
This looks like a common problem with many binary packages.
I don’t have a generic solution yet, but what you really have to do is to move (symlink) the binary file from where it is really installed to the root of your package.
I usually solve this with post-install scripts for my Lambdas. The ones using paramiko run something like:
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so MY_LAMBDA/
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/.libs_cffi_backend/libffi-XXXXXXXX.so.6.0.4 MY_LAMBDA/
ln -s venv/MY_LAMBDA/lib/python3.6/site-packages/nacl MY_LAMBDA/
You will have to identify the exact name of the libffi... file from your virtualenv. It might vary from the machine you compile the library on.
Working on Ubuntu, I haven’t the success with pysftp or paramiko to Lambda.
So I created an EC2 instance (and after an VirtualBox with Amazon Linux 2) on my desktop and develop the same code, with the same libraries. And… that’s works…
may be late, hopefully, will help someone in future. this answer is specific to “_cffi_backend…..”. One other thing that may help is to know how you installed python. if did using brew, you will need to create a virtual env using python. see here:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
i did this in a linux environment without downloading the wheel files that are compatible with various linux distros and it worked
The simplest solution is to use the lambci/lambda container as described here. I wrote about it in some detail here.
In a nutshell, some Python packages (e.g cffi) have OS level dependencies. Because the OS you’re packaging the code on is different to the one Lambda uses, the script fails to run.
You can also get the error No module named '_cffi_backend' when the runtime version of your AWS Lambda function is different than the python version used to create your Lambda Layer.
I received this error when I set the runtime of my Lambda Function to Python 3.10, but I installed the dependencies for my Lambda Layer in an environment that was running Python 3.8.
Using serverless cli with serverless-python-requirements plugin, if you change the AWS Lambda python runtime version and publish, the plugin will use the same pip download cache which was using the old python version. (This cache is outside your git repo, so git clean doesn’t help either.)
That would resulting in having _cffi_backend.cpython-310-x86_64-linux-gnu.so in your lambda, even though you expect to use cpython-311.
The fix is to run serverless requirements cleanCache to delete the pip download cache.
Ran into the same issue while creating a layer for snowflake-python-connector for arm64 using amazonlinux docker container.
About the snowflake-python-connector issue in case it helps someone – packaging the layer using docker on x86 arch gives the following error
{ "errorMessage": "Unable to import module 'lambda_function': /opt/python/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError", "requestId": "07fc4b23-21c2-44e8-a6cd-7b918b84b9f9", "stackTrace": [] }
And using arm64 throws the following error
File "/usr/lib/python2.6/site-packages/cffi/api.py", line 56, in __init__
import _cffi_backend as backend
ImportError: No module named _cffi_backend
Creating the layer as mentioned in the link shared by OzNetNerd under the accepted solution comments worked for me. Posting the link here – https://repost.aws/knowledge-center/lambda-layer-simulated-docker