Stop pip from failing on single package when installing with requirements.txt
Question:
I am installing packages from requirements.txt
pip install -r requirements.txt
The requirements.txt
file reads:
Pillow
lxml
cssselect
jieba
beautifulsoup
nltk
lxml
is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml
fails pip
still runs through and downloads the rest of the packages.
From what I understand the pip install -r requirements.txt
command will fail if any of the packages listed in the requirements.txt
fail to install.
Is there any argument I can pass when running pip install -r requirements.txt
to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?
Answers:
Running each line with pip install
may be a workaround.
cat requirements.txt | xargs -n 1 pip install
Note: -a
parameter is not available under MacOS, so old cat is more portable.
The xargs
solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.
As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends
section of a debian package.
I use the following shell script (requires sed
) to install optional dependencies:
#!/bin/sh
while read dependency; do
dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
# Skip comments
if [[ $dependency_stripped == #* ]]; then
continue
# Skip blank lines
elif [ -z "$dependency_stripped" ]; then
continue
else
if pip install "$dependency_stripped"; then
echo "$dependency_stripped is installed"
else
echo "Could not install $dependency_stripped, skipping"
fi
fi
done < recommends.txt
For Windows:
pip version >=18
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
pip version <18
import sys
import pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
Thanks, Etienne Prothon for windows cases.
But, after upgrading to pip 18, pip package don’t expose main to public. So you may need to change code like this.
# This code install line by line a list of pip package
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.
cat requirements.txt | sed -e '/^s*#.*$/d' -e '/^s*$/d' | xargs -n 1 pip install
Hat tip to this answer for the sed magic.
For Windows:
import os
from pip.__main__ import _main as main
error_log = open('error_log.txt', 'w')
def install(package):
try:
main(['install'] + [str(package)])
except Exception as e:
error_log.write(str(e))
if __name__ == '__main__':
f = open('requirements1.txt', 'r')
for line in f:
install(line)
f.close()
error_log.close()
- Create a local directory, and put your
requirements.txt
file in it.
- Copy the code above and save it as a python file in the same directory. Remember to use
.py
extension, for instance, install_packages.py
- Run this file using a cmd:
python install_packages.py
- All the packages mentioned will be installed in one go without stopping at all. 🙂
You can add other parameters in install function. Like:
main(['install'] + [str(package)] + ['--update'])
For windows users, you can use this:
FOR /F %k in (requirements.txt) DO ( if NOT # == %k ( pip install %k ) )
Logic: for every dependency in file(requirements.txt), install them and ignore those start with "#".
For Windows using PowerShell:
foreach($line in Get-Content requirements.txt) {
if(!($line.StartsWith('#'))){
pip install $line
}
}
One line PowerShell:
Get-Content .requirements.txt | ForEach-Object {pip install $_}
If you need to ignore certain lines then:
Get-Content .requirements.txt | ForEach-Object {if (!$_.startswith("#")){pip install $_}}
OR
Get-Content .requirements.txt | ForEach-Object {if ($_ -notmatch "#"){pip install $_}}
Building on the answer by @MZD, here’s a solution to filter out all text starting with a comment sign #
cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install
Another option is to use pip install --dry-run
to get a list of packages that you need to install and then keep trying it and remove the ones that don’t work.
A very general solution
The following code installs all requirements for:
- multiple requirement files (
requirements1.txt
, requirements2.txt
)
- ignores lines with comments
#
- skips packages, which are not instalable
- runs pip install
each line
(not each word as in some other answers)
$ (cat requirements1.txt; echo ""; cat requirements2.txt) | grep "^[^#]" | xargs -L 1 pip install
I am installing packages from requirements.txt
pip install -r requirements.txt
The requirements.txt
file reads:
Pillow
lxml
cssselect
jieba
beautifulsoup
nltk
lxml
is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml
fails pip
still runs through and downloads the rest of the packages.
From what I understand the pip install -r requirements.txt
command will fail if any of the packages listed in the requirements.txt
fail to install.
Is there any argument I can pass when running pip install -r requirements.txt
to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?
Running each line with pip install
may be a workaround.
cat requirements.txt | xargs -n 1 pip install
Note: -a
parameter is not available under MacOS, so old cat is more portable.
The xargs
solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.
As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends
section of a debian package.
I use the following shell script (requires sed
) to install optional dependencies:
#!/bin/sh
while read dependency; do
dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
# Skip comments
if [[ $dependency_stripped == #* ]]; then
continue
# Skip blank lines
elif [ -z "$dependency_stripped" ]; then
continue
else
if pip install "$dependency_stripped"; then
echo "$dependency_stripped is installed"
else
echo "Could not install $dependency_stripped, skipping"
fi
fi
done < recommends.txt
For Windows:
pip version >=18
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
pip version <18
import sys
import pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
Thanks, Etienne Prothon for windows cases.
But, after upgrading to pip 18, pip package don’t expose main to public. So you may need to change code like this.
# This code install line by line a list of pip package
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
This solution handles empty lines, whitespace lines, # comment lines, whitespace-then-# comment lines in your requirements.txt.
cat requirements.txt | sed -e '/^s*#.*$/d' -e '/^s*$/d' | xargs -n 1 pip install
Hat tip to this answer for the sed magic.
For Windows:
import os
from pip.__main__ import _main as main
error_log = open('error_log.txt', 'w')
def install(package):
try:
main(['install'] + [str(package)])
except Exception as e:
error_log.write(str(e))
if __name__ == '__main__':
f = open('requirements1.txt', 'r')
for line in f:
install(line)
f.close()
error_log.close()
- Create a local directory, and put your
requirements.txt
file in it. - Copy the code above and save it as a python file in the same directory. Remember to use
.py
extension, for instance,install_packages.py
- Run this file using a cmd:
python install_packages.py
- All the packages mentioned will be installed in one go without stopping at all. 🙂
You can add other parameters in install function. Like:
main(['install'] + [str(package)] + ['--update'])
For windows users, you can use this:
FOR /F %k in (requirements.txt) DO ( if NOT # == %k ( pip install %k ) )
Logic: for every dependency in file(requirements.txt), install them and ignore those start with "#".
For Windows using PowerShell:
foreach($line in Get-Content requirements.txt) {
if(!($line.StartsWith('#'))){
pip install $line
}
}
One line PowerShell:
Get-Content .requirements.txt | ForEach-Object {pip install $_}
If you need to ignore certain lines then:
Get-Content .requirements.txt | ForEach-Object {if (!$_.startswith("#")){pip install $_}}
OR
Get-Content .requirements.txt | ForEach-Object {if ($_ -notmatch "#"){pip install $_}}
Building on the answer by @MZD, here’s a solution to filter out all text starting with a comment sign #
cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install
Another option is to use pip install --dry-run
to get a list of packages that you need to install and then keep trying it and remove the ones that don’t work.
A very general solution
The following code installs all requirements for:
- multiple requirement files (
requirements1.txt
,requirements2.txt
) - ignores lines with comments
#
- skips packages, which are not instalable
- runs pip install
each line
(not each word as in some other answers)
$ (cat requirements1.txt; echo ""; cat requirements2.txt) | grep "^[^#]" | xargs -L 1 pip install