AWS Lambda – unable to import module 'lambda_function'
Question:
Like many others before me, I’m trying to run an AWS Lambda function and when I try to test it, I get
“errorMessage”: “Unable to import module ‘lambda_function'”
My Handler is set to lambda_function.lambda_handler, and I indeed have a file named lambda_function.py which contains a function called lambda_handler. Here’s a screenshot as proof:
Everything was working fine when I was writing snippets of code inline in the included IDE, but when I zipped my full program with all of its dependencies and uploaded it, I got the above error.
I’m using the Numpy and Scipy packages, which are quite large. My zipped directory is 34 MB, and my unzipped directory 122 MB. I think this should be fine since the limit is 50 MB for a zipped directory. It appears to be uploading fine, since I see the message:
The deployment package of your Lambda function “one-shot-image-classification” is too large to enable inline code editing. However, you can still invoke your function right now.
I’ve seen that some posts solve this by using virtualenv, but I’m not familiar with that technology and I’m not sure how to use it properly.
I’ve also seen some posts saying that sometimes dependencies have dependencies and I may need to include those, but I’m not sure how to find this out.
Here’s the top portion of lambda_function.py, which should be enough to see the libraries I’m using and that I do indeed have a lambda_handler function:
import os
import boto3
import numpy as np
from scipy.ndimage import imread
from scipy.spatial.distance import cdist
def lambda_handler(event, context):
s3 = boto3.resource('s3')
Here a screenshot of the unzipped version of the directory I’m uploading:
I can also post the policy role that my Lambda is using if that could be an issue.
Any insight is much appreciated!
UPDATE:
Here’s one solution I tried:
1. git clone https://github.com/Miserlou/lambda-packages
2. create a folder in Documents called new_lambda
3. copy my lambda_function.py and the numpy folder from the lambda-packages into new_lambda, along with the scipy library that I compiled using Docker for AWS as per the article: https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/
4. Zip the new_lambda folder by right-clicking it and selecting ‘compress’
My results:
Unable to import module ‘lambda_function’: No module named
‘lambda_function’
To reiterate, my file is named lambda_function.py and contains a function called lambda_handler, which accepts two arguments (as seen above). This information matches that seen in Handler, also seen above.
I am using a Mac computer, if that matters.
UPDATE 2
If I follow the above steps but instead zip the files by directly selecting the files that I want to compress and then right clicking and selecting ‘compress’, I instead get the error
Unable to import module ‘lambda_function’: cannot import name ‘show_config’
Also, the precompiled lambda-packages says that they are compiled for “at least Python 2.7”, but my lambda runtime is 3.6. Could this be an issue?
Answers:
The problem is that your local numpy and pandas are compiled for the local machine’s architecture. Since AWS Lambda uses custom Linux, they are probably not compatible.
So if you want to use them, you have two choices:
-
Compile dependencies on EC2 instance which uses the same Amazon Linux
version as AWS Lambda and create a deployment package.
-
Use one of the precompiled packages from here
P.S. I’ve read comments on a post, so I see that the name of the file and function is ok and numpy is giving you trouble.
The solution was zipping numpy and scipy precompiled packages from this source.
I had a similar issue:
Unable to import module 'lib/lambda_function': No module named 'lib/lambda_function'
The fix for me and possibly for you, was to include a blank __init__.py in the same directory as lambda_function.py.
Why does __init__.py fix the issue?
I understand that it is needed for the directory (lib in my case, . in yours) to be considered a valid Python package.
Here is the reference doc I based that hypothesis on:
5.2.1. Regular packages – Python 3.7.3 documentation
I ran into this issue on MacOS as well. I see you mention the way you select your files affected whether it worked properly. Turns out this is true!
On Mac, if your .DS_Store/MacOS hidden folder sneaks into the directory it seems to break Lambda!
The solution is to
rm .DS_Store
In the deployment zip folder.
I was receiving this same error, but the cause of the error was different. Adding an answer here in case some other weary StackOverflow wanderer finds it helpful.
In my case, I was attempting to upload the below directory where package is a sample python package dependency and my function code is enclosed in lambda_function.py:
I was zipping the entire function directory, which was resulting in the following file structure when deployed to lambda:
In order to run properly, both lambda_function.py and the package directory should be in the top-level lambda directory. In my case ConfigureAppFlow. The function directory is an extra layer that is causing the error.
To fix this, instead of compressing my function directory, I directly compressed the two items inside:
This resulted in the following file structure when deployed to lambda (ignore the _MACOSX folder):
In summary, I’m sure there are a ton of different causes of this issue, but the first thing to check is that the zip file you are uploading to lambda correctly results in a file structure that places lambda_function.py and any dependent packages in your top-level lambda directory.
The way compressing files works seems to be the issue here, I just solved the same problem with my code files. Simply highlight the files/folders you want to compress and right-click and compress that.
For whatever reason, I keep getting an additional folder inside my initial project folder which throw off AWS as it doesn’t know to go 2 directories deep but only looks at the root directory for our files.
Like many others before me, I’m trying to run an AWS Lambda function and when I try to test it, I get
“errorMessage”: “Unable to import module ‘lambda_function'”
My Handler is set to lambda_function.lambda_handler, and I indeed have a file named lambda_function.py which contains a function called lambda_handler. Here’s a screenshot as proof:
Everything was working fine when I was writing snippets of code inline in the included IDE, but when I zipped my full program with all of its dependencies and uploaded it, I got the above error.
I’m using the Numpy and Scipy packages, which are quite large. My zipped directory is 34 MB, and my unzipped directory 122 MB. I think this should be fine since the limit is 50 MB for a zipped directory. It appears to be uploading fine, since I see the message:
The deployment package of your Lambda function “one-shot-image-classification” is too large to enable inline code editing. However, you can still invoke your function right now.
I’ve seen that some posts solve this by using virtualenv, but I’m not familiar with that technology and I’m not sure how to use it properly.
I’ve also seen some posts saying that sometimes dependencies have dependencies and I may need to include those, but I’m not sure how to find this out.
Here’s the top portion of lambda_function.py, which should be enough to see the libraries I’m using and that I do indeed have a lambda_handler function:
import os
import boto3
import numpy as np
from scipy.ndimage import imread
from scipy.spatial.distance import cdist
def lambda_handler(event, context):
s3 = boto3.resource('s3')
Here a screenshot of the unzipped version of the directory I’m uploading:
I can also post the policy role that my Lambda is using if that could be an issue.
Any insight is much appreciated!
UPDATE:
Here’s one solution I tried:
1. git clone https://github.com/Miserlou/lambda-packages
2. create a folder in Documents called new_lambda
3. copy my lambda_function.py and the numpy folder from the lambda-packages into new_lambda, along with the scipy library that I compiled using Docker for AWS as per the article: https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/
4. Zip the new_lambda folder by right-clicking it and selecting ‘compress’
My results:
Unable to import module ‘lambda_function’: No module named
‘lambda_function’
To reiterate, my file is named lambda_function.py and contains a function called lambda_handler, which accepts two arguments (as seen above). This information matches that seen in Handler, also seen above.
I am using a Mac computer, if that matters.
UPDATE 2
If I follow the above steps but instead zip the files by directly selecting the files that I want to compress and then right clicking and selecting ‘compress’, I instead get the error
Unable to import module ‘lambda_function’: cannot import name ‘show_config’
Also, the precompiled lambda-packages says that they are compiled for “at least Python 2.7”, but my lambda runtime is 3.6. Could this be an issue?
The problem is that your local numpy and pandas are compiled for the local machine’s architecture. Since AWS Lambda uses custom Linux, they are probably not compatible.
So if you want to use them, you have two choices:
-
Compile dependencies on EC2 instance which uses the same Amazon Linux
version as AWS Lambda and create a deployment package. -
Use one of the precompiled packages from here
P.S. I’ve read comments on a post, so I see that the name of the file and function is ok and numpy is giving you trouble.
The solution was zipping numpy and scipy precompiled packages from this source.
I had a similar issue:
Unable to import module 'lib/lambda_function': No module named 'lib/lambda_function'
The fix for me and possibly for you, was to include a blank __init__.py in the same directory as lambda_function.py.
Why does __init__.py fix the issue?
I understand that it is needed for the directory (lib in my case, . in yours) to be considered a valid Python package.
Here is the reference doc I based that hypothesis on:
5.2.1. Regular packages – Python 3.7.3 documentation
I ran into this issue on MacOS as well. I see you mention the way you select your files affected whether it worked properly. Turns out this is true!
On Mac, if your .DS_Store/MacOS hidden folder sneaks into the directory it seems to break Lambda!
The solution is to
rm .DS_Store
In the deployment zip folder.
I was receiving this same error, but the cause of the error was different. Adding an answer here in case some other weary StackOverflow wanderer finds it helpful.
In my case, I was attempting to upload the below directory where package is a sample python package dependency and my function code is enclosed in lambda_function.py:
I was zipping the entire function directory, which was resulting in the following file structure when deployed to lambda:
In order to run properly, both lambda_function.py and the package directory should be in the top-level lambda directory. In my case ConfigureAppFlow. The function directory is an extra layer that is causing the error.
To fix this, instead of compressing my function directory, I directly compressed the two items inside:
This resulted in the following file structure when deployed to lambda (ignore the _MACOSX folder):
In summary, I’m sure there are a ton of different causes of this issue, but the first thing to check is that the zip file you are uploading to lambda correctly results in a file structure that places lambda_function.py and any dependent packages in your top-level lambda directory.
The way compressing files works seems to be the issue here, I just solved the same problem with my code files. Simply highlight the files/folders you want to compress and right-click and compress that.
For whatever reason, I keep getting an additional folder inside my initial project folder which throw off AWS as it doesn’t know to go 2 directories deep but only looks at the root directory for our files.