Why is Zappa deploy missing all / most packages present in virtual environment?
Question:
Trying to deploy a python script to AWS via Zappa. Script works in local virtual environment (using virtualenv) but Zappa deploy fails with multiple missing packages. Must be missing something pretty fundamental despite extensive troubleshooting. Details below, any ideas appreciated.
Error: zappa deploy dev errors. zappa tail shows [ERROR] ModuleNotFoundError: No module named 'pandas'. It errors on whatever is the first import statement in main.py` suggesting issue isn’t specific to Pandas pkg.
Screenshot of settings json & project directory structure included below.
Tried the following with no luck:
- Added requirements.txt project directory with all localenv packages shown by
pip freeze
- Added an "include" setting in zappa_settings.json
- Confirmed pandas and other dependency packages appear in zip file produced by
zappa package
EDIT RESOLVED: Project directory size was too big. AWS Lambda (and consequently Zappa) limit project directory including dependencies to 512MB. Exploring alternative cloud infra.
Answers:
RESOLVED: Root cause was project directory size too big. AWS Lambda (and Zappa) limit project directory including dependencies to 512MB by default, and Zappa’s slim_handler doesn’t get around it automatically. So it fails to see the libraries because there isn’t enough space for them on the Lambda at runtime.
This can be fixed in AWS Lambda console, under functions –> [select your function] –> configuration –> general configuration –> ephemeral storage (edit up to directory size).
Saw very little info about this error, so hope this helps someone 🙂
Trying to deploy a python script to AWS via Zappa. Script works in local virtual environment (using virtualenv) but Zappa deploy fails with multiple missing packages. Must be missing something pretty fundamental despite extensive troubleshooting. Details below, any ideas appreciated.
Error: zappa deploy dev errors. zappa tail shows [ERROR] ModuleNotFoundError: No module named 'pandas'. It errors on whatever is the first import statement in main.py` suggesting issue isn’t specific to Pandas pkg.
Screenshot of settings json & project directory structure included below.
Tried the following with no luck:
- Added requirements.txt project directory with all localenv packages shown by
pip freeze - Added an "include" setting in zappa_settings.json
- Confirmed pandas and other dependency packages appear in zip file produced by
zappa package
EDIT RESOLVED: Project directory size was too big. AWS Lambda (and consequently Zappa) limit project directory including dependencies to 512MB. Exploring alternative cloud infra.
RESOLVED: Root cause was project directory size too big. AWS Lambda (and Zappa) limit project directory including dependencies to 512MB by default, and Zappa’s slim_handler doesn’t get around it automatically. So it fails to see the libraries because there isn’t enough space for them on the Lambda at runtime.
This can be fixed in AWS Lambda console, under functions –> [select your function] –> configuration –> general configuration –> ephemeral storage (edit up to directory size).
Saw very little info about this error, so hope this helps someone 🙂