poetry install different package version based on extras
Question:
Using python-poetry, I would like to install different package versions based on the extras that I pass during the installation. E.g. I would like to
# when extra == 'a', install numpy == 1.20.0
$ poetry install -E a
# when extra == 'b', install numpy == 1.19.0
$ poetry install -E b
I tried it with the following toml file
[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["tenticon"]
[tool.poetry.dependencies]
python = "^3.8"
numpy = [
{ version = "1.20.0", markers = "extra == 'a'", optional = true},
{ version = "1.19.0", markers = "extra == 'b'", optional = true}
]
[tool.poetry.extras]
a = [ "numpy" ]
b = [ "numpy" ]
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
but when I do $ poetry install -E a I get
SolverProblemError
Because demo-poetry depends on both numpy (1.20.0) and numpy (1.19.0), version solving failed.
My poetry version is 1.1.6
Answers:
Poetry extras are sets of packages (e.g. a = ["numpy", "scipy"]) that can be optionally installed together with the main dependencies (poetry install -E a). When installing or specifying poetry-built packages, the extras defined in the toml file can be activated as described in PEP-508 definition of extras. Thus, the dependencies required by the a extra could be installed using pip install demo-poetry[a] (just like any other pip extras).
It is indeed possible to use environment markers as install conditions for dependencies (see list of PEP-508 environment markers). However, at the time of writing the environment marker extra is not returned by the relevant function get_marker_env(), and therefore it is ignored as install constraint. The PEP-508 documentation also notes that the extra variable is special, and that there is no current specification for it.
Neither using extras nor groups (from the pre-release 1.2.0a2) was sufficient in order to achieve the expected results. In this regard, I think the comment from @finswimmer is correct: groups are not mutually exclusive, but poetry checks that the dependency resolution works in every case.
The closest I could get to an acceptable solution was by defining conditions based on the python version, or based on the platform. Note that these conditions should always be mutually exclusive, otherwise you will get an error.
For instance, if you have the following pyproject.toml:
[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["vreyespue"]
[tool.poetry.dependencies]
numpy = [
{ version = "1.19.0", python = "~3.7"},
{ version = "1.20.0", python = "~3.9"}
]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
You could install different versions of numpy in different environments:
$ poetry env use 3.7
Using virtualenv: /***/demo-poetry-***-py3.7
$ poetry install
Installing numpy (1.19.0)
$ poetry env use 3.9
Using virtualenv: /***/demo-poetry-***-py3.9
$ poetry install
Installing numpy (1.20.0)
This bothered me for internal development, and while it was extremely frustrating since neither poetry nor cleo offer any documentation on their SDK, I finally managed to create a plugin for this (pypi, git).
The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without or --only flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.
Installation via poetry add poet-plugin (if starting from an empty, clean project, without such dependencies), or then via:
poetry shell
pip install poet-plugin
I have not battle-tested this, but it works nicely for our use case with e.g.
poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod
Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.
Using python-poetry, I would like to install different package versions based on the extras that I pass during the installation. E.g. I would like to
# when extra == 'a', install numpy == 1.20.0
$ poetry install -E a
# when extra == 'b', install numpy == 1.19.0
$ poetry install -E b
I tried it with the following toml file
[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["tenticon"]
[tool.poetry.dependencies]
python = "^3.8"
numpy = [
{ version = "1.20.0", markers = "extra == 'a'", optional = true},
{ version = "1.19.0", markers = "extra == 'b'", optional = true}
]
[tool.poetry.extras]
a = [ "numpy" ]
b = [ "numpy" ]
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
but when I do $ poetry install -E a I get
SolverProblemError
Because demo-poetry depends on both numpy (1.20.0) and numpy (1.19.0), version solving failed.
My poetry version is 1.1.6
Poetry extras are sets of packages (e.g. a = ["numpy", "scipy"]) that can be optionally installed together with the main dependencies (poetry install -E a). When installing or specifying poetry-built packages, the extras defined in the toml file can be activated as described in PEP-508 definition of extras. Thus, the dependencies required by the a extra could be installed using pip install demo-poetry[a] (just like any other pip extras).
It is indeed possible to use environment markers as install conditions for dependencies (see list of PEP-508 environment markers). However, at the time of writing the environment marker extra is not returned by the relevant function get_marker_env(), and therefore it is ignored as install constraint. The PEP-508 documentation also notes that the extra variable is special, and that there is no current specification for it.
Neither using extras nor groups (from the pre-release 1.2.0a2) was sufficient in order to achieve the expected results. In this regard, I think the comment from @finswimmer is correct: groups are not mutually exclusive, but poetry checks that the dependency resolution works in every case.
The closest I could get to an acceptable solution was by defining conditions based on the python version, or based on the platform. Note that these conditions should always be mutually exclusive, otherwise you will get an error.
For instance, if you have the following pyproject.toml:
[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["vreyespue"]
[tool.poetry.dependencies]
numpy = [
{ version = "1.19.0", python = "~3.7"},
{ version = "1.20.0", python = "~3.9"}
]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
You could install different versions of numpy in different environments:
$ poetry env use 3.7
Using virtualenv: /***/demo-poetry-***-py3.7
$ poetry install
Installing numpy (1.19.0)
$ poetry env use 3.9
Using virtualenv: /***/demo-poetry-***-py3.9
$ poetry install
Installing numpy (1.20.0)
This bothered me for internal development, and while it was extremely frustrating since neither poetry nor cleo offer any documentation on their SDK, I finally managed to create a plugin for this (pypi, git).
The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without or --only flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.
Installation via poetry add poet-plugin (if starting from an empty, clean project, without such dependencies), or then via:
poetry shell
pip install poet-plugin
I have not battle-tested this, but it works nicely for our use case with e.g.
poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod
Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.