Building lxml for Python 2.7 on Windows

Question:

I am trying to build lxml for Python 2.7 on Windows 64 bit machine. I couldn’t find lxml egg for Python 2.7 version. So I am compiling it from sources. I am following instructions on this site

http://lxml.de/build.html

under static linking section. I am getting error

C:Documents and SettingsAdministratorDesktoplxmlpackagelxml-2.2.6lxml-2.2.
6>python setup.py bdist_wininst --static
Building lxml version 2.2.6.
NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' need
s to be available.
ERROR: 'xslt-config' is not recognized as an internal or external command,
operable program or batch file.

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt
Building against libxml2/libxslt in one of the following directories:
  ..libxml2-2.7.6--win32--w2k--x64lib
  ..libxslt-1.1.26--win32--w2k--x64--0002lib
  ..zlib-1.2.4--win32--w2k--x64
  ..iconv-1.9.1--win32--w2k--x64-0001lib
running bdist_wininst
running build
running build_py
running build_ext
building 'lxml.etree' extension
error: Unable to find vcvarsall.bat

Can any one help me with this? I tried setting the path to have Microsoft Visual Studio..
I can run vcvarsall.bat from the commandline.. but python is having problems

Asked By: Kamal

||
Source

Answers:

I bet you’re not using VS 2008 for this 🙂

There’s def find_vcvarsall(version): function (guess what, it looks for vcvarsall.bat) in distutils with the following comment

At first it tries to find the
productdir of VS 2008 in the registry.
If that fails it falls back to the
VS90COMNTOOLS env var.

If you’re not using VS 2008 then you have neither the registry key nor suitable environment variable and that’s why distutils can’t find vcvarsall.bat file. It does not check if the bat file is reachable through the PATH environment variable.

The solution is to define VS90COMNTOOLS variable to point to Tools directory of Visual Studio.

That being said take a look at 11.4. distutils.msvccompiler — Microsoft Compiler section in Python’s docs which states

Typically, extension modules need to
be compiled with the same compiler
that was used to compile Python.

Martin v. Loewis in the email titled Download Visual Studio Express 2008 now on python-list mailing list states the same

Python 2.6, 2.7, and 3.1 are all built
with that release (i.e. 2008). Because
of another long tradition, Python
extension modules must be built with
the same compiler version (more
specifically, CRT version) as Python
itself. So to build extension modules
for any of these releases, you need to
have a copy of VS 2008 or VS 2008
Express.

In the light of above statements you should use VS 2008 if you want to build lxml for Python 2.7 so although setting VS90COMNTOOLS takes care of finding vcvarsall.bat file it’s not the solution.

That being said 🙂 people do try to use older CRT with newer compiler:
Can I use Visual Studio 2010’s C++ compiler with Visual Studio 2008’s C++ Runtime Library?
How to Enforce C++ compiler to use specific CRT version?
VS 2008 – Link against older C runtime

I’d like to thank Kev Dwyer (for pointing out importance of version of VS which is used) and Stefan Behnel (for pointing me to distutils as a place dealing with compiler’s configuration) in the thread Problem building lxml under Windows – error: Unable to find vcvarsall.bat on lxml mailing list. I’d like to also thank agronholm from freenode #distutils IRC channel for confirmation that distutils does contain code which looks for vcvarsall.bat file.

Answered By: Piotr Dobrogost

After following the recommended solution:

  1. downloading VCForPython27.msi from Microsoft,
  2. installing it (Win7, Python(x, y) 2.7.9 32bit),
  3. entering / updating the environment variable VS90COMNTOOLS to
    the installation directory value (C:Program Files (x86)Common
    FilesMicrosoftVisual C++ for Python9.0)

my problem still existed (want to build a Python extension in C).

I had to do the following 2 unbelievably dirty tweaks, before everything now is indeed working:

  1. modify “msvc9compiler.py” in “C:Python27Libdistutils”,
    function find_vcvarsall, to now point to “Visual C++ for
    Python”     instead of to “VC”.
  2. copy the directories founder under “C:Program Files (x86)Common
    FilesMicrosoftVisual C++ for Python9.0”     to “C:Program Files
    (x86)Common FilesMicrosoftVisual C++ for Python”     (i.e. one dir
    level up).

I cannot tell who was doing something wrong here – probably I.

EDIT. Moving directories works because of the issue described in this distutils bug.

even if VS90COMNTOOLS is set, msvc9compiler isn’t able to find vcvarsall.bat because it is installed in %installdir%/vcvarsall.bat and not %installdir%/VC/vcvarsall.bat

The described workaround is using the Visual C++ command prompt:

  1. Enter MSVC for Python command prompt

  2. SET DISTUTILS_USE_SDK=1

  3. SET MSSdk=1

  4. python.exe setup.py …

Answered By: Jorj McKie

Jorj McKie was almost correct: indeed installing VCForPython27.msi isn’t enough, and yes there is an issue in distutils which prevent it from finding find_vcvarsall. In fact the issue is not directly in distutils, but in how VCForPython27.msi was packaged and where vcvarsall.bat is placed (the folders layout is different from the VS2008 SDK).

A simple workaround meanwhile this gets patched maybe in Python 2.7.11: use setuptools instead of distutils.

Another manual workaround if you’re stuck with distutils:

1) Enter MSVC for Python command prompt
2) SET DISTUTILS_USE_SDK=1
3) SET MSSdk=1
4) you can then build your C extensions: python.exe setup.py ...

Bug report and workaround by Gregory Szorc:
http://bugs.python.org/issue23246

More info and a workaround for using %%cython magic inside IPython: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows

Answered By: gaborous

https://github.com/develersrl/gccwinbinaries

I had similar issues. This worked instantly without anything other than using an install wizard and setting one preference.

Answered By: Robbie Capps