Cannot install Scrapy on macOS M1 13.0.1 – lxml installation error: '/usr/bin/clang' failed with exit code 1

Question:

I’m trying to install Scrapy in my Macbook M1 with macOS Ventura 13.0.1, but it is throwing an error while trying to install lxml.

Installing collected packages: lxml, jmespath, itemadapter, idna, filelock, cssselect, charset-normalizer, cffi, certifi, attrs, requests, parsel, hyperlink, cryptography, Automat, Twisted, service-identity, requests-file, pyOpenSSL, itemloaders, tldextract, scrapy
  Running setup.py install for lxml ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for lxml did not run successfully.
  │ exit code: 1
  ╰─> [115 lines of output]
      Building lxml version 4.9.2.
      Building without Cython.
      Building against libxml2 2.9.13 and libxslt 1.1.35
      running install
      /Users/tomasazevedo/Documents/py_scripts/webscraping_test/env/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-39
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/_elementpath.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/sax.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/builder.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/doctestcompare.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/usedoctest.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/cssselect.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/ElementInclude.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/soupparser.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/defs.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/clean.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/html5parser.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/formfill.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/builder.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      copying src/lxml/html/diff.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/html
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron
      copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron
      copying src/lxml/etree.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/etree_api.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/lxml.etree.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/lxml.etree_api.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/objectify.pyx -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/etree.pyx -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/public-api.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xmlid.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/cleanup.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xslt.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xpath.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/debug.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/serializer.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/classlookup.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/saxparser.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/objectpath.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/relaxng.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xinclude.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/iterparse.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/parser.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/dtd.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/apihelpers.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xmlerror.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/readonlytree.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/extensions.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xmlschema.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/schematron.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/docloader.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/nsclasses.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/parsertarget.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/xsltext.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/proxy.pxi -> build/lib.macosx-10.9-universal2-cpython-39/lxml
      copying src/lxml/includes/xmlerror.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/c14n.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/xmlschema.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/__init__.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/schematron.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/tree.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/uri.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/etreepublic.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/xpath.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/htmlparser.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/xslt.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/config.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/xmlparser.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/xinclude.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/dtdvalid.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/relaxng.pxd -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/lxml-version.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      copying src/lxml/includes/etree_defs.h -> build/lib.macosx-10.9-universal2-cpython-39/lxml/includes
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/rng
      copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/rng
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl
      creating build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.9-universal2-cpython-39/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      running build_ext
      building 'lxml.etree' extension
      creating build/temp.macosx-10.9-universal2-cpython-39
      creating build/temp.macosx-10.9-universal2-cpython-39/src
      creating build/temp.macosx-10.9-universal2-cpython-39/src/lxml
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/tomasazevedo/Documents/py_scripts/webscraping_test/env/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -c src/lxml/etree.c -o build/temp.macosx-10.9-universal2-cpython-39/src/lxml/etree.o -w -flat_namespace
      src/lxml/etree.c:96:10: fatal error: 'Python.h' file not found
      #include "Python.h"
               ^~~~~~~~~~
      1 error generated.
      Compile failed: command '/usr/bin/clang' failed with exit code 1
      cc -I/usr/include/libxml2 -c /var/folders/zv/tjy58slj5mb96l_9g6x105r00000gn/T/xmlXPathInitbd2u4ada.c -o var/folders/zv/tjy58slj5mb96l_9g6x105r00000gn/T/xmlXPathInitbd2u4ada.o
      cc var/folders/zv/tjy58slj5mb96l_9g6x105r00000gn/T/xmlXPathInitbd2u4ada.o -lxml2 -o a.out
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lxml

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I’ve tried a bunch of things to try to install lxml on its own, but to no avail, the error is always the same.

I have the most recent Python version 3.9.6

Does anyone have any idea how to solve this?

Thanks in advance.

Asked By: tomasaazevedo

||

Answers:

For some reason was able to install lxml and scrapy outside the virtual environment.

Answered By: tomasaazevedo

had the same issue in a venv using 3.9; upgraded to 3.10 and it resolved

Answered By: M Pokorny
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.