Check if my Python has all required packages
Question:
I have a requirements.txt file with a list of packages that are required for my virtual environment. Is it possible to find out whether all the packages mentioned in the file are present. If some packages are missing, how to find out which are the missing packages?
Answers:
You can run pip freeze to see what you have installed and compare it to your requirements.txt file.
If you want to install missing modules you can run pip install -r requirements.txt and that will install any missing modules and tell you at the end which ones were missing and installed.
UPDATE:
An up-to-date and improved way to do this is via distutils.text_file.TextFile. See Acumenus’ answer below for details.
ORIGINAL:
The pythonic way of doing it is via the pkg_resources API. The requirements are written in a format understood by setuptools. E.g:
Werkzeug>=0.6.1
Flask
Django>=1.3
The example code:
import pkg_resources
from pkg_resources import DistributionNotFound, VersionConflict
# dependencies can be any iterable with strings,
# e.g. file line-by-line iterator
dependencies = [
'Werkzeug>=0.6.1',
'Flask>=0.9',
]
# here, if a dependency is not met, a DistributionNotFound or VersionConflict
# exception is thrown.
pkg_resources.require(dependencies)
Based on the answer by Zaur, assuming you indeed use a requirements file, you may want a unit test, perhaps in tests/test_requirements.py, that confirms the availability of packages.
Moreover, this approach uses a subtest to independently confirm each requirement. This is useful so that all failures are documented. Without subtests, only a single failure is documented.
"""Test availability of required packages."""
import unittest
from pathlib import Path
import pkg_resources
_REQUIREMENTS_PATH = Path(__file__).parent.with_name("requirements.txt")
class TestRequirements(unittest.TestCase):
"""Test availability of required packages."""
def test_requirements(self):
"""Test that each required package is available."""
# Ref: https://stackoverflow.com/a/45474387/
requirements = pkg_resources.parse_requirements(_REQUIREMENTS_PATH.open())
for requirement in requirements:
requirement = str(requirement)
with self.subTest(requirement=requirement):
pkg_resources.require(requirement)
If you’re interested in doing this from the command line, pip-missing-reqs will list missing packages. Example:
$ pip-missing-reqs directory
Missing requirements:
directory/exceptions.py:11 dist=grpcio module=grpc
(pip check and pipdeptree --warn fail only audit installed packages for compatibility with each other, without checking requirements.txt.)
In addition to check whether modules listed in requirements.txt are installed, you may want to check whether all used modules by your project are indeed listed in the requirements.txt file.
If you are using flake8 for style-checking, you can add flake8-requirements plugin for flake8. It will automatically check whether imported modules are available in setup.py, requirements.txt or pyproject.toml file. Additionally, for custom modules you can add custom configuration with known-modules (in order to prevent flake8 warnings). For more configuration options, see flake8-requirements project’s description.
You can use the -r option from pip freeze that verifies that. It generates a WARNING log for packages that are not installed. One appropriated verbose mode should be selected in order the WARNING message to be shown. For example:
$ pip -vvv freeze -r requirements.txt | grep "not installed"
WARNING: Requirement file [requirements.txt] contains six==1.15.0, but package 'six' is not installed
Here’s a one-liner based on Zaur Nasibov’s answer for if you don’t care to know which packages are not installed:
python3 -c "import pkg_resources; pkg_resources.require(open('requirements.txt',mode='r'))" &>/dev/null
Whether the command finishes successfully can then be used to do a pip install.
As an equivalent to Ruby’s bundle check || bundle install, we’re doing:
python3 -c "import pkg_resources; pkg_resources.require(open('requirements.txt',mode='r'))" &>/dev/null || pip3 install --ignore-installed -r requirements.txt
I realise this is not addressing the exact question, but this page comes up first when Googling for that. And anyway, you’d only really be wanting to know what the missing dependencies are in order to then satisfy them.
We can’t just use pip3 check for this, since it does not look at the requirements.txt
Running
pip freeze -r requirements.txt
it will compare installed ones to the requirements file and warn you which ones are not
I have a requirements.txt file with a list of packages that are required for my virtual environment. Is it possible to find out whether all the packages mentioned in the file are present. If some packages are missing, how to find out which are the missing packages?
You can run pip freeze to see what you have installed and compare it to your requirements.txt file.
If you want to install missing modules you can run pip install -r requirements.txt and that will install any missing modules and tell you at the end which ones were missing and installed.
UPDATE:
An up-to-date and improved way to do this is via distutils.text_file.TextFile. See Acumenus’ answer below for details.
ORIGINAL:
The pythonic way of doing it is via the pkg_resources API. The requirements are written in a format understood by setuptools. E.g:
Werkzeug>=0.6.1
Flask
Django>=1.3
The example code:
import pkg_resources
from pkg_resources import DistributionNotFound, VersionConflict
# dependencies can be any iterable with strings,
# e.g. file line-by-line iterator
dependencies = [
'Werkzeug>=0.6.1',
'Flask>=0.9',
]
# here, if a dependency is not met, a DistributionNotFound or VersionConflict
# exception is thrown.
pkg_resources.require(dependencies)
Based on the answer by Zaur, assuming you indeed use a requirements file, you may want a unit test, perhaps in tests/test_requirements.py, that confirms the availability of packages.
Moreover, this approach uses a subtest to independently confirm each requirement. This is useful so that all failures are documented. Without subtests, only a single failure is documented.
"""Test availability of required packages."""
import unittest
from pathlib import Path
import pkg_resources
_REQUIREMENTS_PATH = Path(__file__).parent.with_name("requirements.txt")
class TestRequirements(unittest.TestCase):
"""Test availability of required packages."""
def test_requirements(self):
"""Test that each required package is available."""
# Ref: https://stackoverflow.com/a/45474387/
requirements = pkg_resources.parse_requirements(_REQUIREMENTS_PATH.open())
for requirement in requirements:
requirement = str(requirement)
with self.subTest(requirement=requirement):
pkg_resources.require(requirement)
If you’re interested in doing this from the command line, pip-missing-reqs will list missing packages. Example:
$ pip-missing-reqs directory
Missing requirements:
directory/exceptions.py:11 dist=grpcio module=grpc
(pip check and pipdeptree --warn fail only audit installed packages for compatibility with each other, without checking requirements.txt.)
In addition to check whether modules listed in requirements.txt are installed, you may want to check whether all used modules by your project are indeed listed in the requirements.txt file.
If you are using flake8 for style-checking, you can add flake8-requirements plugin for flake8. It will automatically check whether imported modules are available in setup.py, requirements.txt or pyproject.toml file. Additionally, for custom modules you can add custom configuration with known-modules (in order to prevent flake8 warnings). For more configuration options, see flake8-requirements project’s description.
You can use the -r option from pip freeze that verifies that. It generates a WARNING log for packages that are not installed. One appropriated verbose mode should be selected in order the WARNING message to be shown. For example:
$ pip -vvv freeze -r requirements.txt | grep "not installed"
WARNING: Requirement file [requirements.txt] contains six==1.15.0, but package 'six' is not installed
Here’s a one-liner based on Zaur Nasibov’s answer for if you don’t care to know which packages are not installed:
python3 -c "import pkg_resources; pkg_resources.require(open('requirements.txt',mode='r'))" &>/dev/null
Whether the command finishes successfully can then be used to do a pip install.
As an equivalent to Ruby’s bundle check || bundle install, we’re doing:
python3 -c "import pkg_resources; pkg_resources.require(open('requirements.txt',mode='r'))" &>/dev/null || pip3 install --ignore-installed -r requirements.txt
I realise this is not addressing the exact question, but this page comes up first when Googling for that. And anyway, you’d only really be wanting to know what the missing dependencies are in order to then satisfy them.
We can’t just use pip3 check for this, since it does not look at the requirements.txt
Running
pip freeze -r requirements.txt
it will compare installed ones to the requirements file and warn you which ones are not