PIP Constraints Files
Question:
I was reading the docs and couldn’t wrap my head around it:
Constraints files are requirements files that only control which
version of a requirement is installed, not whether it is installed or
not. Their syntax and contents is nearly identical to Requirements
Files.
There is one key difference: Including a package in a constraints file
does not trigger installation of the package.
- So does that mean I need to requirement files first then run constraint?
- Need to have both requirements.txt and constraints.txt?
- Or just
-c requirements.txt?
Could someone please explain in plain English that what the new PIP feature: Constraints Files does?
Answers:
I haven’t used constrains files myself, so I may be wrong, but here’s how I understand it from the documentation.
Let’s say, you have multiple projects which you install using pip. Some of them depend on a popular package foobar directly, some indirectly (via their dependencies), some don’t depend on it at all.
One day you decide to change foobar code a bit, to solve some problem related to your environment, so you go ahead and create a local copy of foobar. Now you want to ensure that all your projects use this local copy, but you don’t want to spend time figuring out which of them depend on foobar and then editing their requirements.txt or setup.py files.
Here comes constraints file, where you point out to your local copy of foobar and use it for all your projects, not caring which of them need foobar, because constrains file only applies when the project being installed actually depends on foobar (whether directly or indirectly).
There’s an even better example of possible usage on reddit.
Update: pip 20.3, released Nov 30, 2020, introduced a new resolver that fixes some design issues. It also reimplements the constraints feature. It is now difficult or impossible to use the feature the way I describe below. I do not understand the new constraints implementation, and I no longer use it. I’ve had success with pip-compile from pip-tools. I specify top-level requirements in requirements.in, and pip-compile generates a requirements.txt with specific versions an package hashes. See requirements.in and requirements.txt in the ichnaea project for a working example.
Original answer below, for pip 20.2 and previous
I think a constraints file is a good way to keep your "true" requirements separate from your full install list.
It’s good practice to fully specify package versions in your requirements file. For example, if you are installing django-allauth with Django LTS, pin it to the latest releases (as of my answer):
Django==1.8.12
django-allauth==0.25.2
When you install the package, it ends up installing some required packages as well. So, you add those as well, so that everyone gets the same versions of packages:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5
requests==2.9.1
requests-oauthlib==0.6.1
And then you get the bug report "Doesn’t work under Python 3". Oops, python-openid is Python 2 only, and python3-openid is used instead, further requiring defusedxml:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5 ; python_version < '3.0'
python3-openid==3.0.10 ; python_version >= '3.0'
defusedxml==0.4.1 ; python_version >= '3.0'
requests==2.9.1
requests-oauthlib==0.6.1
Now requirements.txt is getting ugly, and it’s hard to see the "requirements" of Django and django-allauth in the mess.
Here is a requirements.txt that refers to a constraints file:
-c constraints.txt
Django==1.8.12
django-allauth==0.25.2
And constraints.txt with a helpful comment:
# django-allauth requirements
oauthlib==1.0.3
python-openid==2.2.5
python3-openid==3.0.10
defusedxml==0.4.1
requests==2.9.1
requests-oauthlib==0.6.1
No Python classifiers are needed, because constraints are only installed if a package requires them, and are ignored otherwise. Additionally, if a package stops requiring another package 2 years down the road, fresh installs will stop installing it.
I think this, plus some comments, is a useful way to communicate what packages you are using for the project, and which ones are included because they are dependencies.
I think it gets even more useful if you are using pip 8.x’s hash-checking mode, which requires specifying versions for dependencies-of-dependencies. If you go down that path, I recommend hashin to help you manage the hashes. See browsercompat for a complicated requirements setup using constraints and hashes.
From the book “Secret Recipes of the Python Ninja”
Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed but provide no control over the actual installation.
Let say you have requirements.txt file with following code
# requirements.txt
pandas
and constraints.txt
# constraints.txt
# math / science / graph stuff
bokeh==0.11.1
numpy==1.10.4
pandas==0.17.1
scipy==0.17.0
openpyxl==2.3.3
patsy==0.4.1
matplotlib==1.5.1
ggplot==0.6.8
seaborn==0.7.0
scikit-learn==0.17
executing
pip install -c constraints.txt
will install all packages from requirements.txt and using constraints.txt file for version constraint.
Constraint files are very useful when you provide production environments with optimized package compilations for the productive infrastructure.
We provide such environments in docker containers with versions of numpy, scipy, tensorflow, opencv, … optimized for our production servers. In order to ensure that the requirements files developers set up to make the environment reproducible for them don’t overwrite these optimized installations, we use constraint files. They are written as part of the build process. pip is made aware of the constraints by setting the PIP_CONSTRAINT environment variable.
Before, developers always had to make sure their requirements line up with what’s available in the prod containers, which can be a pain given that some minor version numbers iterate quite quickly.
I was reading the docs and couldn’t wrap my head around it:
Constraints files are requirements files that only control which
version of a requirement is installed, not whether it is installed or
not. Their syntax and contents is nearly identical to Requirements
Files.There is one key difference: Including a package in a constraints file
does not trigger installation of the package.
- So does that mean I need to requirement files first then run constraint?
- Need to have both requirements.txt and constraints.txt?
- Or just
-c requirements.txt?
Could someone please explain in plain English that what the new PIP feature: Constraints Files does?
I haven’t used constrains files myself, so I may be wrong, but here’s how I understand it from the documentation.
Let’s say, you have multiple projects which you install using pip. Some of them depend on a popular package foobar directly, some indirectly (via their dependencies), some don’t depend on it at all.
One day you decide to change foobar code a bit, to solve some problem related to your environment, so you go ahead and create a local copy of foobar. Now you want to ensure that all your projects use this local copy, but you don’t want to spend time figuring out which of them depend on foobar and then editing their requirements.txt or setup.py files.
Here comes constraints file, where you point out to your local copy of foobar and use it for all your projects, not caring which of them need foobar, because constrains file only applies when the project being installed actually depends on foobar (whether directly or indirectly).
There’s an even better example of possible usage on reddit.
Update: pip 20.3, released Nov 30, 2020, introduced a new resolver that fixes some design issues. It also reimplements the constraints feature. It is now difficult or impossible to use the feature the way I describe below. I do not understand the new constraints implementation, and I no longer use it. I’ve had success with pip-compile from pip-tools. I specify top-level requirements in requirements.in, and pip-compile generates a requirements.txt with specific versions an package hashes. See requirements.in and requirements.txt in the ichnaea project for a working example.
Original answer below, for pip 20.2 and previous
I think a constraints file is a good way to keep your "true" requirements separate from your full install list.
It’s good practice to fully specify package versions in your requirements file. For example, if you are installing django-allauth with Django LTS, pin it to the latest releases (as of my answer):
Django==1.8.12
django-allauth==0.25.2
When you install the package, it ends up installing some required packages as well. So, you add those as well, so that everyone gets the same versions of packages:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5
requests==2.9.1
requests-oauthlib==0.6.1
And then you get the bug report "Doesn’t work under Python 3". Oops, python-openid is Python 2 only, and python3-openid is used instead, further requiring defusedxml:
Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5 ; python_version < '3.0'
python3-openid==3.0.10 ; python_version >= '3.0'
defusedxml==0.4.1 ; python_version >= '3.0'
requests==2.9.1
requests-oauthlib==0.6.1
Now requirements.txt is getting ugly, and it’s hard to see the "requirements" of Django and django-allauth in the mess.
Here is a requirements.txt that refers to a constraints file:
-c constraints.txt
Django==1.8.12
django-allauth==0.25.2
And constraints.txt with a helpful comment:
# django-allauth requirements
oauthlib==1.0.3
python-openid==2.2.5
python3-openid==3.0.10
defusedxml==0.4.1
requests==2.9.1
requests-oauthlib==0.6.1
No Python classifiers are needed, because constraints are only installed if a package requires them, and are ignored otherwise. Additionally, if a package stops requiring another package 2 years down the road, fresh installs will stop installing it.
I think this, plus some comments, is a useful way to communicate what packages you are using for the project, and which ones are included because they are dependencies.
I think it gets even more useful if you are using pip 8.x’s hash-checking mode, which requires specifying versions for dependencies-of-dependencies. If you go down that path, I recommend hashin to help you manage the hashes. See browsercompat for a complicated requirements setup using constraints and hashes.
From the book “Secret Recipes of the Python Ninja”
Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed but provide no control over the actual installation.
Let say you have requirements.txt file with following code
# requirements.txt
pandas
and constraints.txt
# constraints.txt
# math / science / graph stuff
bokeh==0.11.1
numpy==1.10.4
pandas==0.17.1
scipy==0.17.0
openpyxl==2.3.3
patsy==0.4.1
matplotlib==1.5.1
ggplot==0.6.8
seaborn==0.7.0
scikit-learn==0.17
executing
pip install -c constraints.txt
will install all packages from requirements.txt and using constraints.txt file for version constraint.
Constraint files are very useful when you provide production environments with optimized package compilations for the productive infrastructure.
We provide such environments in docker containers with versions of numpy, scipy, tensorflow, opencv, … optimized for our production servers. In order to ensure that the requirements files developers set up to make the environment reproducible for them don’t overwrite these optimized installations, we use constraint files. They are written as part of the build process. pip is made aware of the constraints by setting the PIP_CONSTRAINT environment variable.
Before, developers always had to make sure their requirements line up with what’s available in the prod containers, which can be a pain given that some minor version numbers iterate quite quickly.