Why can't I get `pip install lxml` to work within a virtualenv?
Question:
Note: I’m using virtualenvwrapper.
Before activating the virtual environment:
$ pip install lxml
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib/python2.7/dist-packages
Cleaning up...
After activating the virtual environment:
(test-env)$ pip install lxml
force/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error:
libxml/xmlversion.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /home/chaz/dev/envs/test-with-system-python-force/bin/python2
.7 -c "import setuptools;__file__='/home/chaz/dev/envs/test-with-
system-python-force/build/lxml/setup.py';exec(compile(open(__file__).
read().replace('rn', 'n'), __file__, 'exec'))" install --record
/tmp/pip-bJ6Q_B-record/install-record.txt --single-version-externally
-managed --install-headers /home/chaz/dev/envs/test-env/include/site/python2.7 failed with error code 1 in
/home/chaz/dev/envs/test-env/build/lxml
Storing complete log in /home/chaz/.pip/pip.log
Answers:
You probably already have lxml installed on your system, perhaps installed due to a system package. Thus, the first attempt (pip install lxml without an active virtualenv) doesn’t fail, but it also doesn’t install it; it really doesn’t do anything.
In a virtualenv, by default, the system packages are ignored. Therefore, pip thinks that lxml is not installed. Therefore, it tries to install it into your virtual environment.
lxml contains C modules that need to be compiled in order to install properly. However, the compilation of those C modules rely on your having some “development libraries” already installed as well. These development libraries are C libraries, not Python, and as such pip won’t be able to automatically fetch them from the internet and install them for you.
Therefore, you will need to install these development libraries on your own, most likely using your package manager. In a Debian system (like Ubuntu), this is…
apt-get install libxml2-dev libxslt-dev
This will install the libxml2 and libxslt development libraries to your local system. If you try again to install lxml, the C module compilation step should work because now these development libraries are on your system.
The error message you were receiving was due to the fact that these libraries were missing (the libxml/xmlversion.h: No such file or directory part of the error message).
See also: How to install lxml on Ubuntu
for centos users:
when getting:
error: command ‘gcc’ failed with exit status 1
DO:
sudo yum install libxslt-devel libxml2-devel
If you have lxml installed at the system level, and want to migrate it into a virtualenv that you didn’t create with --system-site-packages, you can symlink it into your virtualenv‘s dist-packages folder.
Outside your virtualenv, in a python shell:
import lxml
print lxml.__file__
In my case, it’s found in /usr/lib/python2.7/dist-packages. There’ll be an lxml folder and egg-info file. Wherever your virtualenv is, go into its /lib/python-x.y/dist-packages folder (you may need to create dist-packages), and symlink both the library folder and egg into it.
You’re most likely looking for this:
Microsoft Visual C++ 14.0 is required (Unable to find vcvarsall.bat)
Look for the visual studio website and go into:
“Tools for visual studio” at the bottom, expand it by clicking.
Select Download next to “Build Tools for Visual Studio 2017” at the top.
Note: I’m using virtualenvwrapper.
Before activating the virtual environment:
$ pip install lxml
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib/python2.7/dist-packages
Cleaning up...
After activating the virtual environment:
(test-env)$ pip install lxml
force/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error:
libxml/xmlversion.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /home/chaz/dev/envs/test-with-system-python-force/bin/python2
.7 -c "import setuptools;__file__='/home/chaz/dev/envs/test-with-
system-python-force/build/lxml/setup.py';exec(compile(open(__file__).
read().replace('rn', 'n'), __file__, 'exec'))" install --record
/tmp/pip-bJ6Q_B-record/install-record.txt --single-version-externally
-managed --install-headers /home/chaz/dev/envs/test-env/include/site/python2.7 failed with error code 1 in
/home/chaz/dev/envs/test-env/build/lxml
Storing complete log in /home/chaz/.pip/pip.log
You probably already have lxml installed on your system, perhaps installed due to a system package. Thus, the first attempt (pip install lxml without an active virtualenv) doesn’t fail, but it also doesn’t install it; it really doesn’t do anything.
In a virtualenv, by default, the system packages are ignored. Therefore, pip thinks that lxml is not installed. Therefore, it tries to install it into your virtual environment.
lxml contains C modules that need to be compiled in order to install properly. However, the compilation of those C modules rely on your having some “development libraries” already installed as well. These development libraries are C libraries, not Python, and as such pip won’t be able to automatically fetch them from the internet and install them for you.
Therefore, you will need to install these development libraries on your own, most likely using your package manager. In a Debian system (like Ubuntu), this is…
apt-get install libxml2-dev libxslt-dev
This will install the libxml2 and libxslt development libraries to your local system. If you try again to install lxml, the C module compilation step should work because now these development libraries are on your system.
The error message you were receiving was due to the fact that these libraries were missing (the libxml/xmlversion.h: No such file or directory part of the error message).
See also: How to install lxml on Ubuntu
for centos users:
when getting:
error: command ‘gcc’ failed with exit status 1
DO:
sudo yum install libxslt-devel libxml2-devel
If you have lxml installed at the system level, and want to migrate it into a virtualenv that you didn’t create with --system-site-packages, you can symlink it into your virtualenv‘s dist-packages folder.
Outside your virtualenv, in a python shell:
import lxml
print lxml.__file__
In my case, it’s found in /usr/lib/python2.7/dist-packages. There’ll be an lxml folder and egg-info file. Wherever your virtualenv is, go into its /lib/python-x.y/dist-packages folder (you may need to create dist-packages), and symlink both the library folder and egg into it.
You’re most likely looking for this:
Microsoft Visual C++ 14.0 is required (Unable to find vcvarsall.bat)
Look for the visual studio website and go into:
“Tools for visual studio” at the bottom, expand it by clicking.
Select Download next to “Build Tools for Visual Studio 2017” at the top.