Upgrading setuptools on OSX El Capitan
Question:
I’m trying to upgrade setuptools. Well actually I’m trying to upgrade ansible but it’s trying to upgrade setuptools and failing. Trying to do it myself also fails. Even trying to uninstall it fails
$ sudo -H pip install --upgrade setuptools
Collecting setuptools
Using cached setuptools-18.4-py2.py3-none-any.whl
Installing collected packages: setuptools
Found existing installation: setuptools 1.1.6
Uninstalling setuptools-1.1.6:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/commands/install.py", line 311, in run
root=options.root_path,
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_install.py", line 716, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/utils/__init__.py", line 315, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
copytree(src, real_dst, symlinks=True)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]
I haven’t the foggiest idea what’s wrong. Looking at all the files in /System/Library/Frameworks/Python.framework/Versions/2.7/ and below every single file is owned by root:wheel
How can I fix it?
Answers:
This happens due to the System Integrity Protection introduced in OS X El Capitan.
Adding --user python to the command allowed this to work.
In other words
pip install --upgrade setuptools --user python
The answer is that you cannot update setuptools on OSX for the factory python that ships with the OS. The reason is that the files in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python cannot be removed or modified by the user. Not only the normal user but even if you have root permissions you will be unable to modify those files.
Sure you can use various other methods to install a newer version of setuptools but this will not override the default system package. Meaning if you go the suggested route of use the --user python flag, or you choose to install them without root into ~/Library, these will not override the system files version.
The reason you or root are unable to modify the system defaults is due to SIP permission restrictions in El Capitan+. You could disable SIP but this is generally not recommended.
Instead the only reasonable solution is to use a python virtualenv.
Overview
The problem is conflicting Python libraries, combined with System Integrity Protection (SIP) on Mac OS which protects the system Python libraries.
The best solution in my opinion is to uninstall and re-install your own Python installation and keep that separate to the Python libraries provided by Mac OS which are protected by SIP.
I favour this to disabling SIP as I expect SIP to be a part of any future Mac OS releases and SIP is not the cause here, it just exposes the problem of conflicting Python libraries.
Details
I also had this issue trying to install ansible.
My problem started when I followed the ansible installation instructions for Mac OS, which is to install via pip and to install pip with easy_install as described at Latest Releases Via Pip
Problem is when installing pip this way, the easy_install is the Mac OS provided easy_install at /usr/bin/easy_install but it writes to the easy-install.pth file at /Library/Python/2.7/site-packages/easy-install.pth and that file references the Mac OS provided Python libraries.
Subsequently installing ansible with pip install ansible then reports that the setuptools requirement is satisfied by the Mac OS provided Python library at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Ansible requires setuptools but doesn’t specify a version. Another dependent package, paramiko, requires setuptools >= 11.3, but this doesn’t appear to be checked when installing ansible via pip. When you try to run ansible though it complains that setup tools is 1.1.6 This is the version of setuptools provided with Mac OS, and is now protected by SIP, so it can’t be upgraded.
Rather than disable SIP, I fixed this by uninstalling my local Python as hinted at at https://docs.python.org/2.7/using/mac.html#getting-and-installing-macpython
and then downloading and installing again.
If you’re concerned about the rm, you can mv instead
To uninstall I did
sudo rm -rf /Library/Python
sudo rm -rf /Applications/Python 2.7/
sudo rm /usr/local/bin/ansible # executable
sudo rm /usr/local/bin/python* # symlinks to /Library/Python/2.7
sudo rm /usr/local/bin/easy_install*
# and so on for references to /Library/Python/2.7 in /usr/local/bin
I then downloaded the 2.7.13 installer package for Mac OS X from https://www.python.org/downloads/
and installed it.
This installed a local python and pip at /Library/Frameworks/Python.framework/Versions/2.7 and symlinks in /usr/local/bin as well as prepending /Library/Frameworks/Python.framework/Versions/2.7 to my $PATH. This keeps everything separate from the Mac OS provided libraries at /System/Library/Frameworks/Python.framework and /usr/bin
so that I get
which pip
/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
I then installed ansible with pip install ansible
Then when I run which ansible I get
/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible
and in pip list
setuptools (28.8.0)
six (1.10.0)
and ansible now works for me because the Mac OS Python libraries are not referenced by the new install.
Note that because of the installer setting $PATH, which python is now /Library/Frameworks/Python.framework/Versions/2.7/bin/python and /usr/local/bin/python is a symlink to this.
If you want the system Python you’ll have to use /usr/bin/python or change the $PATH
I’m trying to upgrade setuptools. Well actually I’m trying to upgrade ansible but it’s trying to upgrade setuptools and failing. Trying to do it myself also fails. Even trying to uninstall it fails
$ sudo -H pip install --upgrade setuptools
Collecting setuptools
Using cached setuptools-18.4-py2.py3-none-any.whl
Installing collected packages: setuptools
Found existing installation: setuptools 1.1.6
Uninstalling setuptools-1.1.6:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/commands/install.py", line 311, in run
root=options.root_path,
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_install.py", line 716, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/utils/__init__.py", line 315, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
copytree(src, real_dst, symlinks=True)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-OyTXsR-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]
I haven’t the foggiest idea what’s wrong. Looking at all the files in /System/Library/Frameworks/Python.framework/Versions/2.7/ and below every single file is owned by root:wheel
How can I fix it?
This happens due to the System Integrity Protection introduced in OS X El Capitan.
Adding --user python to the command allowed this to work.
In other words
pip install --upgrade setuptools --user python
The answer is that you cannot update setuptools on OSX for the factory python that ships with the OS. The reason is that the files in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python cannot be removed or modified by the user. Not only the normal user but even if you have root permissions you will be unable to modify those files.
Sure you can use various other methods to install a newer version of setuptools but this will not override the default system package. Meaning if you go the suggested route of use the --user python flag, or you choose to install them without root into ~/Library, these will not override the system files version.
The reason you or root are unable to modify the system defaults is due to SIP permission restrictions in El Capitan+. You could disable SIP but this is generally not recommended.
Instead the only reasonable solution is to use a python virtualenv.
Overview
The problem is conflicting Python libraries, combined with System Integrity Protection (SIP) on Mac OS which protects the system Python libraries.
The best solution in my opinion is to uninstall and re-install your own Python installation and keep that separate to the Python libraries provided by Mac OS which are protected by SIP.
I favour this to disabling SIP as I expect SIP to be a part of any future Mac OS releases and SIP is not the cause here, it just exposes the problem of conflicting Python libraries.
Details
I also had this issue trying to install ansible.
My problem started when I followed the ansible installation instructions for Mac OS, which is to install via pip and to install pip with easy_install as described at Latest Releases Via Pip
Problem is when installing pip this way, the easy_install is the Mac OS provided easy_install at /usr/bin/easy_install but it writes to the easy-install.pth file at /Library/Python/2.7/site-packages/easy-install.pth and that file references the Mac OS provided Python libraries.
Subsequently installing ansible with pip install ansible then reports that the setuptools requirement is satisfied by the Mac OS provided Python library at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Ansible requires setuptools but doesn’t specify a version. Another dependent package, paramiko, requires setuptools >= 11.3, but this doesn’t appear to be checked when installing ansible via pip. When you try to run ansible though it complains that setup tools is 1.1.6 This is the version of setuptools provided with Mac OS, and is now protected by SIP, so it can’t be upgraded.
Rather than disable SIP, I fixed this by uninstalling my local Python as hinted at at https://docs.python.org/2.7/using/mac.html#getting-and-installing-macpython
and then downloading and installing again.
If you’re concerned about the rm, you can mv instead
To uninstall I did
sudo rm -rf /Library/Python
sudo rm -rf /Applications/Python 2.7/
sudo rm /usr/local/bin/ansible # executable
sudo rm /usr/local/bin/python* # symlinks to /Library/Python/2.7
sudo rm /usr/local/bin/easy_install*
# and so on for references to /Library/Python/2.7 in /usr/local/bin
I then downloaded the 2.7.13 installer package for Mac OS X from https://www.python.org/downloads/
and installed it.
This installed a local python and pip at /Library/Frameworks/Python.framework/Versions/2.7 and symlinks in /usr/local/bin as well as prepending /Library/Frameworks/Python.framework/Versions/2.7 to my $PATH. This keeps everything separate from the Mac OS provided libraries at /System/Library/Frameworks/Python.framework and /usr/bin
so that I get
which pip
/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
I then installed ansible with pip install ansible
Then when I run which ansible I get
/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible
and in pip list
setuptools (28.8.0)
six (1.10.0)
and ansible now works for me because the Mac OS Python libraries are not referenced by the new install.
Note that because of the installer setting $PATH, which python is now /Library/Frameworks/Python.framework/Versions/2.7/bin/python and /usr/local/bin/python is a symlink to this.
If you want the system Python you’ll have to use /usr/bin/python or change the $PATH