How to create a deployable Python Lamba zip using Poetry
Question:
I’ve been spending a few days trying to figure out how best to build a Python Lambda bundle when using Poetry. I found a few blogs that that outline the same technique but those didn’t work in my situation. The solution provided in the blogs is to use pip install to install the needed dependencies into a specific directory and zip it up.
poetry run pip install -t dist/lambda .
cd dist/lambda
zip -r ../lambda.zip .
However, this doesn’t work if you use path dependencies with Poetry. You get an error from pip stating pip._vendor.pkg_resources.RequirementParseError: Invalid URL: for any local dependency.
I did run into the Poetry Bundle Plugin and it looked promising. Using it did work in that it installed the needed dependencies and the project itself into the chosen target directory.
poetry self add poetry-plugin-bundle
poetry bundle venv .venv-lambda
cd .venv-lambda/lib/python*/site-packages/
zip -r ../../../../dist/lambda.zip .
The problem with this approach is that it installs more than just the mainline dependencies, but also the dev and test dependencies. There is no option to specify which dependency group to include or exclude. There is an open issue with a PR that is waiting to be merged to resolve this. Once that is resolved, this is likely the ideal solution.
Until then, I need something different/better.
Answers:
Ultimately I found this documentation from AWS for how to create a lambda archive from a Python virtual environment. Using Poetry’s install command, I was able to install just the main runtime dependencies into the Poetry projects created virtual environment, including any local path based dependencies. However, this doesn’t install the project itself so the source code needs to be copied in before being archived. In my case, I use a dedicated source directory/module for my code.
poetry install --only main --sync
mkdir -p dist/lambda-package
cp --recursive .venv/lib/python*/site-packages/* dist/lambda-package/
cp --recursive my_project_source_directory dist/lambda-package/
cd dist/lambda-package
zip -r ../dist/lambda.zip .
The above commands are what I use on my CI build. The local .venv directory is used because the following Poetry setting virtualenvs.in-project is set to true.
The other thing necessary for this to work is to not use editable path based dependencies, or at least just do that locally during development. Marking them as editable will not install the dependency into the virtual environment, but will rather just create a link to the project source code. This will not get picked up when creating the zip file.
This isn’t perfect as there is likely more that gets bundled than necessary but it does remove any dev and test dependencies from the Poetry plugin solution. Also, because on my CI build server, I cache the installed dependencies in the virtual environment, this means at the end of my build, none of the dev or test dependencies are present to be cached and get installed on every run.
I hope this helps someone else in a similar situation.
Here’s how I’ve been doing it:
rm -rf dist package
poetry install --only main --sync
poetry build
poetry run pip install --upgrade -t package dist/*.whl
cd package; mkdir -p out; zip -r -q out/mylambda.zip . -x '*.pyc'
I’ve been spending a few days trying to figure out how best to build a Python Lambda bundle when using Poetry. I found a few blogs that that outline the same technique but those didn’t work in my situation. The solution provided in the blogs is to use pip install to install the needed dependencies into a specific directory and zip it up.
poetry run pip install -t dist/lambda .
cd dist/lambda
zip -r ../lambda.zip .
However, this doesn’t work if you use path dependencies with Poetry. You get an error from pip stating pip._vendor.pkg_resources.RequirementParseError: Invalid URL: for any local dependency.
I did run into the Poetry Bundle Plugin and it looked promising. Using it did work in that it installed the needed dependencies and the project itself into the chosen target directory.
poetry self add poetry-plugin-bundle
poetry bundle venv .venv-lambda
cd .venv-lambda/lib/python*/site-packages/
zip -r ../../../../dist/lambda.zip .
The problem with this approach is that it installs more than just the mainline dependencies, but also the dev and test dependencies. There is no option to specify which dependency group to include or exclude. There is an open issue with a PR that is waiting to be merged to resolve this. Once that is resolved, this is likely the ideal solution.
Until then, I need something different/better.
Ultimately I found this documentation from AWS for how to create a lambda archive from a Python virtual environment. Using Poetry’s install command, I was able to install just the main runtime dependencies into the Poetry projects created virtual environment, including any local path based dependencies. However, this doesn’t install the project itself so the source code needs to be copied in before being archived. In my case, I use a dedicated source directory/module for my code.
poetry install --only main --sync
mkdir -p dist/lambda-package
cp --recursive .venv/lib/python*/site-packages/* dist/lambda-package/
cp --recursive my_project_source_directory dist/lambda-package/
cd dist/lambda-package
zip -r ../dist/lambda.zip .
The above commands are what I use on my CI build. The local .venv directory is used because the following Poetry setting virtualenvs.in-project is set to true.
The other thing necessary for this to work is to not use editable path based dependencies, or at least just do that locally during development. Marking them as editable will not install the dependency into the virtual environment, but will rather just create a link to the project source code. This will not get picked up when creating the zip file.
This isn’t perfect as there is likely more that gets bundled than necessary but it does remove any dev and test dependencies from the Poetry plugin solution. Also, because on my CI build server, I cache the installed dependencies in the virtual environment, this means at the end of my build, none of the dev or test dependencies are present to be cached and get installed on every run.
I hope this helps someone else in a similar situation.
Here’s how I’ve been doing it:
rm -rf dist package
poetry install --only main --sync
poetry build
poetry run pip install --upgrade -t package dist/*.whl
cd package; mkdir -p out; zip -r -q out/mylambda.zip . -x '*.pyc'