How to list the name of all Poetry extras?
Question:
I’m working on a project where I need to install all of the Poetry extras to run an automated build. What is the easiest way to get that list as a space-separated list, for use in poetry install --extras="$extras"?
Some possibilities:
- I could use the Python toml package, but that would require setting up a separate virtualenv to install it.
- A quick
sed command to get the tool.poetry.extras section of pyproject.toml and pull out the names would be a hack, and only works if the TOML is nicely and consistently formatted.
- Downloading and installing a third party CLI tool just for this seems overkill; I’d prefer a
sed hack over that.
- Maybe there’s a commonly installed tool (as in, already installed in the official Ubuntu 20.04 Docker image) which can do this or something adjacent like converting TOML to JSON for processing with a more well-known tool like
jq, but I’m not aware of it.
Answers:
The current hacky solution: The TOML file is formatted using pretty-format-toml, so the structure is completely uniform – an extra basically looks like this:
content_iterator = [
"jsonschema",
"pynamodb",
]
Based on that I’ve been able to use this command to get all the extras as a space-separated list:
echo "PYTHON_EXTRAS=$(sed --quiet '/[tool.poetry.extras]/,/^[/{s/^(.*) = [/1/p}' pyproject.toml
| sed --null-data 's/n/ /g;s/ $//')"
| tee -a $GITHUB_ENV
Basically, within the tool.poetry.extras section look for any line with = [ and print the stuff before that. Then replace newlines with spaces. Finally remove the trailing space (poetry is picky, and will refuse to install a list of extras with trailing spaces or newlines between names) and put the result into the relevant configuration file.
This is extremely ugly and brittle, so I’m just posting this for anyone else who needs a solution right now.
Sorry, cannot add as a comment, but probably you could use now
poetry install --all-extras
with newer Poetry version https://python-poetry.org/docs/pyproject/#extras
I’m working on a project where I need to install all of the Poetry extras to run an automated build. What is the easiest way to get that list as a space-separated list, for use in poetry install --extras="$extras"?
Some possibilities:
- I could use the Python toml package, but that would require setting up a separate virtualenv to install it.
- A quick
sedcommand to get thetool.poetry.extrassection of pyproject.toml and pull out the names would be a hack, and only works if the TOML is nicely and consistently formatted. - Downloading and installing a third party CLI tool just for this seems overkill; I’d prefer a
sedhack over that. - Maybe there’s a commonly installed tool (as in, already installed in the official Ubuntu 20.04 Docker image) which can do this or something adjacent like converting TOML to JSON for processing with a more well-known tool like
jq, but I’m not aware of it.
The current hacky solution: The TOML file is formatted using pretty-format-toml, so the structure is completely uniform – an extra basically looks like this:
content_iterator = [
"jsonschema",
"pynamodb",
]
Based on that I’ve been able to use this command to get all the extras as a space-separated list:
echo "PYTHON_EXTRAS=$(sed --quiet '/[tool.poetry.extras]/,/^[/{s/^(.*) = [/1/p}' pyproject.toml
| sed --null-data 's/n/ /g;s/ $//')"
| tee -a $GITHUB_ENV
Basically, within the tool.poetry.extras section look for any line with = [ and print the stuff before that. Then replace newlines with spaces. Finally remove the trailing space (poetry is picky, and will refuse to install a list of extras with trailing spaces or newlines between names) and put the result into the relevant configuration file.
This is extremely ugly and brittle, so I’m just posting this for anyone else who needs a solution right now.
Sorry, cannot add as a comment, but probably you could use now
poetry install --all-extras
with newer Poetry version https://python-poetry.org/docs/pyproject/#extras