How to specify install order for python pip?
Question:
I’m working with fabric(0.9.4)+pip(0.8.2) and I need to install some python modules for multiple servers. All servers have old version of setuptools (0.6c8) which needs to be upgraded for pymongo module. Pymongo requires setuptools>=0.6c9.
My problem is that pip starts installation with pymongo instead of setuptools which causes pip to stop. Shuffling module order in requirements file doesn’t seem to help.
requirements.txt:
setuptools>=0.6c9
pymongo==1.9
simplejson==2.1.3
Is there a way to specify install order for pip as it doesn’t seem to do it properly by itself?
This can be resolved with two separate requirements files but it would be nice if I didn’t need to maintain multiple requirements files now or in the future.
Problem persists with pip 0.8.3.
Answers:
This is a silly hack, but might just work. Write a bash script that reads from your requirements file line by line and runs the pip command on it.
#!/bin/bash
for line in $(cat requirements.txt)
do
pip install $line -E /path/to/virtualenv
done
Sorry, my first answer was wrong, because I had setuptools>=0.6c9.
It seems it is not possible because pymongo’s setup.py needs setuptools>=0.6c9, but pip has only downloaded setuptools>=0.6c9, and not installed yet.
Someone discussed about it in the issue I pointed before.
I have my own created an issue some weeks ago about it: Do not run egg_info to each package in requirements list before installing the previous packages.
Sorry for the noisy.
First answer:
Upgrade your pip to 0.8.3 version, it has a bugfix to installation order.
Now if you upgrade everything works 🙂
Check the news here: http://www.pip-installer.org/en/0.8.3/news.html
I ended up running pip inside virtualenv instead of using “pip -E” because with -E pip could still see servers site-packages and that obviously messed up some of the installs.
I also had trouble with servers without virtualenvs. Even if I installed setuptools with separate pip command pymongo would refuse to be installed.
I resolved this by installing setuptools separately with easy_install as this seems to be problem between pip and setuptools.
snippets from fabfile.py:
env.activate = "source %s/bin/activate" % virtualenv_path
_virtualenv("easy_install -U setuptools")
_virtualenv("pip install -r requirements.txt")
def _virtualenv(command)
if env.virtualenv:
sudo(env.activate + "&&" + command)
else:
sudo(command)
I had these problems with pip 0.8.3 and 0.8.2.
You can just use:
cat requirements.txt | xargs pip install
Sadly the upgrade suggestion won’t work. If you read the other details in https://github.com/pypa/pip/issues/24 you will see why
pip will build all packages first, before attempting to install them. So with a requirements file like the following
numpy==1.7.1
scipy==0.13.2
statsmodels==0.5.0
The build of statsmodels will fail with the following statement
ImportError: statsmodels requires numpy
The workaround given for manually calling pip for each entry in the requirements file (via a shell script) seems to be the only current solution.
Following on from @lukasrms’s solution – I had to do this to get pip to install my requirements one-at-a-time:
cat requirements.txt | xargs -n 1 pip install
To allow all types of entries (for example packages from git repositories) in requirements.txt you need to use the following set of commands
cat requirements.txt | xargs -n 1 -L 1 pip install
-n 1 and -L 1 options are necessary to install packages one by one and treat every line in the requirements.txt file as a separate item.
If you have comments in your requirements file you’ll want to use:
grep -v "^#" requirements.txt | xargs pip install
Pymongo requires setuptools>=0.6c9
How do you know? Requires to build or to install? You don’t say what version of Pymongo you were trying to install but looking at setup.py file for current (3.2.2) version there’s no specification of neither what Pymongo requires to run setup.py (setup_requires) nor what it requires to install (install_requires). With no such information pip can’t ensure specific version of setuptools. If Pymongo requires specific version of setuptools to run its setup.py (as opposed to requiring setuptools to run setup function itself) then the other problem is that until recently there was no way to specify this. Now there’s specification – PEP 518 – Specifying Minimum Build System Requirements for Python Projects, which should be shortly implemented in pip – Implement PEP 518 support #3691.
As to order of installation, this was fixed in pip 6.1.0;
From pip install – Installation Order section of pip’s documentation:
As of v6.1.0, pip installs dependencies before their dependents, i.e.
in “topological order”. This is the only commitment pip currently
makes related to order.
And later:
Prior to v6.1.0, pip made no commitments about install order.
However, without proper specification of requirements by Pymongo it won’t help either.
Here is what worked nicely for me:
for name in "^numpy" "^scipy" "^cython" "^scikit"
do
grep -r -h $name $requirements_dir | awk '{print($1)}' | xargs pip install
done
With that solution you don’t modify the original requirement files, but you attempt to install packages in a correct order in advance.
I’m working with fabric(0.9.4)+pip(0.8.2) and I need to install some python modules for multiple servers. All servers have old version of setuptools (0.6c8) which needs to be upgraded for pymongo module. Pymongo requires setuptools>=0.6c9.
My problem is that pip starts installation with pymongo instead of setuptools which causes pip to stop. Shuffling module order in requirements file doesn’t seem to help.
requirements.txt:
setuptools>=0.6c9
pymongo==1.9
simplejson==2.1.3
Is there a way to specify install order for pip as it doesn’t seem to do it properly by itself?
This can be resolved with two separate requirements files but it would be nice if I didn’t need to maintain multiple requirements files now or in the future.
Problem persists with pip 0.8.3.
This is a silly hack, but might just work. Write a bash script that reads from your requirements file line by line and runs the pip command on it.
#!/bin/bash
for line in $(cat requirements.txt)
do
pip install $line -E /path/to/virtualenv
done
Sorry, my first answer was wrong, because I had setuptools>=0.6c9.
It seems it is not possible because pymongo’s setup.py needs setuptools>=0.6c9, but pip has only downloaded setuptools>=0.6c9, and not installed yet.
Someone discussed about it in the issue I pointed before.
I have my own created an issue some weeks ago about it: Do not run egg_info to each package in requirements list before installing the previous packages.
Sorry for the noisy.
First answer:
Upgrade your pip to 0.8.3 version, it has a bugfix to installation order.
Now if you upgrade everything works 🙂
Check the news here: http://www.pip-installer.org/en/0.8.3/news.html
I ended up running pip inside virtualenv instead of using “pip -E” because with -E pip could still see servers site-packages and that obviously messed up some of the installs.
I also had trouble with servers without virtualenvs. Even if I installed setuptools with separate pip command pymongo would refuse to be installed.
I resolved this by installing setuptools separately with easy_install as this seems to be problem between pip and setuptools.
snippets from fabfile.py:
env.activate = "source %s/bin/activate" % virtualenv_path
_virtualenv("easy_install -U setuptools")
_virtualenv("pip install -r requirements.txt")
def _virtualenv(command)
if env.virtualenv:
sudo(env.activate + "&&" + command)
else:
sudo(command)
I had these problems with pip 0.8.3 and 0.8.2.
You can just use:
cat requirements.txt | xargs pip install
Sadly the upgrade suggestion won’t work. If you read the other details in https://github.com/pypa/pip/issues/24 you will see why
pip will build all packages first, before attempting to install them. So with a requirements file like the following
numpy==1.7.1
scipy==0.13.2
statsmodels==0.5.0
The build of statsmodels will fail with the following statement
ImportError: statsmodels requires numpy
The workaround given for manually calling pip for each entry in the requirements file (via a shell script) seems to be the only current solution.
Following on from @lukasrms’s solution – I had to do this to get pip to install my requirements one-at-a-time:
cat requirements.txt | xargs -n 1 pip install
To allow all types of entries (for example packages from git repositories) in requirements.txt you need to use the following set of commands
cat requirements.txt | xargs -n 1 -L 1 pip install
-n 1 and -L 1 options are necessary to install packages one by one and treat every line in the requirements.txt file as a separate item.
If you have comments in your requirements file you’ll want to use:
grep -v "^#" requirements.txt | xargs pip install
Pymongo requires setuptools>=0.6c9
How do you know? Requires to build or to install? You don’t say what version of Pymongo you were trying to install but looking at setup.py file for current (3.2.2) version there’s no specification of neither what Pymongo requires to run setup.py (setup_requires) nor what it requires to install (install_requires). With no such information pip can’t ensure specific version of setuptools. If Pymongo requires specific version of setuptools to run its setup.py (as opposed to requiring setuptools to run setup function itself) then the other problem is that until recently there was no way to specify this. Now there’s specification – PEP 518 – Specifying Minimum Build System Requirements for Python Projects, which should be shortly implemented in pip – Implement PEP 518 support #3691.
As to order of installation, this was fixed in pip 6.1.0;
From pip install – Installation Order section of pip’s documentation:
As of v6.1.0, pip installs dependencies before their dependents, i.e.
in “topological order”. This is the only commitment pip currently
makes related to order.
And later:
Prior to v6.1.0, pip made no commitments about install order.
However, without proper specification of requirements by Pymongo it won’t help either.
Here is what worked nicely for me:
for name in "^numpy" "^scipy" "^cython" "^scikit"
do
grep -r -h $name $requirements_dir | awk '{print($1)}' | xargs pip install
done
With that solution you don’t modify the original requirement files, but you attempt to install packages in a correct order in advance.