tell pip to install the dependencies of packages listed in a requirement file
Question:
Developing a Django web app, I have a list of packages I need to install in a virtualenv. Say:
Django==1.3.1
--extra-index-url=http://dist.pinaxproject.com/dev/
Pinax==0.9b1.dev10
git+git://github.com/pinax/pinax-theme-bootstrap.git@cff4f5bbe9f87f0c67ee9ada9aa8ae82978f9890
# and other packages
Initially I installed them manually, one by one, along the development. This installed the required dependencies and I finally used pip freeze before deploying the app.
Problem is, as I upgraded some packages, some dependencies are no longer used nor required but they keep being listed by pip freeze.
Now, I’d like to set up a new virtualenv this way:
- put the needed packages (without their dependencies) in a requirement file,
like manual-requirements.txt
- install them with their dependencies
pip install -r manual-requirement.txt (← problem, this does not install the dependencies)
- freeze the full virtualenv
pip freeze -r manual-requirements.txt > full-requirements.txt
and deploy.
Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I’d like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.
edit: actually, pip does install dependencies not explicitly listed in the requirement file, even if the documentation tells us that such files are flat. I was wrong about which dependencies i expected to be installed. I’ll let this question for anyone in doubt about pip not installing all dependencies.
Answers:
Given your comment to the question (where you say that executing the install for a single package works as expected), I would suggest looping over your requirement file. In bash:
#!/bin/sh
while read p; do
pip install $p
done < requirements.pip
HTH!
Simply use:
pip install -r requirements.txt
This installs all dependencies listed in requirements.txt file.
Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I’d like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.
That’s what pip-tools package is for (from https://github.com/jazzband/pip-tools):
Installation
$ pip install --upgrade pip # pip-tools needs pip==6.1 or higher (!)
$ pip install pip-tools
Example usage for pip-compile
Suppose you have a Flask project, and want to pin it for production. Write the following line to a file:
# requirements.in
Flask
Now, run pip-compile requirements.in:
$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# Make changes in requirements.in, then run this to update:
#
# pip-compile requirements.in
#
flask==0.10.1
itsdangerous==0.24 # via flask
jinja2==2.7.3 # via flask
markupsafe==0.23 # via jinja2
werkzeug==0.10.4 # via flask
And it will produce your requirements.txt, with all the Flask dependencies (and all underlying dependencies) pinned. Put this file under version control as well and periodically re-run pip-compile to update the packages.
Example usage for pip-sync
Now that you have a requirements.txt, you can use pip-sync to update your virtual env to reflect exactly what’s in there. Note: this will install/upgrade/uninstall everything necessary to match the requirements.txt contents.
$ pip-sync
Uninstalling flake8-2.4.1:
Successfully uninstalled flake8-2.4.1
Collecting click==4.1
Downloading click-4.1-py2.py3-none-any.whl (62kB)
100% |████████████████████████████████| 65kB 1.8MB/s
Found existing installation: click 4.0
Uninstalling click-4.0:
Successfully uninstalled click-4.0
Successfully installed click-4.1
Extending Piotr’s answer, if you also need a way to figure what to put in requirements.in, you can first use pip-chill to find the minimal set of required packages you have. By combining these tools, you can show the dependency reason why each package is installed. The full cycle looks like this:
- Create virtual environment:
$ python3 -m venv venv
- Activate it:
$ . venv/bin/activate
- Install newest version of pip, pip-tools and pip-chill:
(venv)$ pip install --upgrade pip
(venv)$ pip install pip-tools pip-chill
- Build your project, install more pip packages, etc, until you want to save…
- Extract minimal set of packages (ie, top-level without dependencies):
(venv)$ pip-chill --no-version > requirements.in
- Compile list of all required packages (showing dependency reasons):
(venv)$ pip-compile requirements.in
- Make sure the current installation is synchronized with the list:
(venv)$ pip-sync
As @Ming mentioned:
pip install -r file.txt
Here’s a simple line to force update all dependencies:
while read -r package; do pip install --upgrade --force-reinstall $package;done < pipfreeze.txt
or
pip list --outdated --format=freeze | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U
Developing a Django web app, I have a list of packages I need to install in a virtualenv. Say:
Django==1.3.1
--extra-index-url=http://dist.pinaxproject.com/dev/
Pinax==0.9b1.dev10
git+git://github.com/pinax/pinax-theme-bootstrap.git@cff4f5bbe9f87f0c67ee9ada9aa8ae82978f9890
# and other packages
Initially I installed them manually, one by one, along the development. This installed the required dependencies and I finally used pip freeze before deploying the app.
Problem is, as I upgraded some packages, some dependencies are no longer used nor required but they keep being listed by pip freeze.
Now, I’d like to set up a new virtualenv this way:
- put the needed packages (without their dependencies) in a requirement file,
likemanual-requirements.txt - install them with their dependencies
pip install -r manual-requirement.txt(← problem, this does not install the dependencies) - freeze the full virtualenv
pip freeze -r manual-requirements.txt > full-requirements.txt
and deploy.
Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I’d like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.
edit: actually, pip does install dependencies not explicitly listed in the requirement file, even if the documentation tells us that such files are flat. I was wrong about which dependencies i expected to be installed. I’ll let this question for anyone in doubt about pip not installing all dependencies.
Given your comment to the question (where you say that executing the install for a single package works as expected), I would suggest looping over your requirement file. In bash:
#!/bin/sh
while read p; do
pip install $p
done < requirements.pip
HTH!
Simply use:
pip install -r requirements.txt
This installs all dependencies listed in requirements.txt file.
Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I’d like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.
That’s what pip-tools package is for (from https://github.com/jazzband/pip-tools):
Installation
$ pip install --upgrade pip # pip-tools needs pip==6.1 or higher (!)
$ pip install pip-tools
Example usage for pip-compile
Suppose you have a Flask project, and want to pin it for production. Write the following line to a file:
# requirements.in
Flask
Now, run pip-compile requirements.in:
$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# Make changes in requirements.in, then run this to update:
#
# pip-compile requirements.in
#
flask==0.10.1
itsdangerous==0.24 # via flask
jinja2==2.7.3 # via flask
markupsafe==0.23 # via jinja2
werkzeug==0.10.4 # via flask
And it will produce your requirements.txt, with all the Flask dependencies (and all underlying dependencies) pinned. Put this file under version control as well and periodically re-run pip-compile to update the packages.
Example usage for pip-sync
Now that you have a requirements.txt, you can use pip-sync to update your virtual env to reflect exactly what’s in there. Note: this will install/upgrade/uninstall everything necessary to match the requirements.txt contents.
$ pip-sync
Uninstalling flake8-2.4.1:
Successfully uninstalled flake8-2.4.1
Collecting click==4.1
Downloading click-4.1-py2.py3-none-any.whl (62kB)
100% |████████████████████████████████| 65kB 1.8MB/s
Found existing installation: click 4.0
Uninstalling click-4.0:
Successfully uninstalled click-4.0
Successfully installed click-4.1
Extending Piotr’s answer, if you also need a way to figure what to put in requirements.in, you can first use pip-chill to find the minimal set of required packages you have. By combining these tools, you can show the dependency reason why each package is installed. The full cycle looks like this:
- Create virtual environment:
$ python3 -m venv venv - Activate it:
$ . venv/bin/activate - Install newest version of pip, pip-tools and pip-chill:
(venv)$ pip install --upgrade pip
(venv)$ pip install pip-tools pip-chill - Build your project, install more pip packages, etc, until you want to save…
- Extract minimal set of packages (ie, top-level without dependencies):
(venv)$ pip-chill --no-version > requirements.in - Compile list of all required packages (showing dependency reasons):
(venv)$ pip-compile requirements.in - Make sure the current installation is synchronized with the list:
(venv)$ pip-sync
As @Ming mentioned:
pip install -r file.txt
Here’s a simple line to force update all dependencies:
while read -r package; do pip install --upgrade --force-reinstall $package;done < pipfreeze.txt
or
pip list --outdated --format=freeze | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U