File not found error when pip installing python-lzo
Question:
I’m trying to install python-lzo on mac OS Sierra. I have used brew to install the c lzo library:
brew update
brew install lzo
I then try and install python-lzo using a virtual environment and pip:
$ virtualenv ./env
$ . ./env/bin/activate
$ pip install python-lzo
Which gives me the following output:
Collecting python-lzo
Using cached python-lzo-1.11.tar.gz
Building wheels for collected packages: python-lzo
Running setup.py bdist_wheel for python-lzo: started
Running setup.py bdist_wheel for python-lzo: finished with status 'error'
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-wDRYjx/python-lzo/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" bdist_wheel -d /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/tmp_ueOY4pip-wheel- --python-tag cp27:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running bdist_wheel
running build
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
lzomodule.c:35:10: fatal error: 'lzo1x.h' file not found
#include <lzo1x.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
Running setup.py clean for python-lzo
Failed to build python-lzo
Installing collected packages: python-lzo
Running setup.py install for python-lzo: started
Running setup.py install for python-lzo: finished with status 'error'
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-wDRYjx/python-lzo/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" install --record /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-8RUWxo-record/install-record.txt --single-version-externally-managed --compile:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running install
running build
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
lzomodule.c:35:10: fatal error: 'lzo1x.h' file not found
#include <lzo1x.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
The lzo1x.h file does exist here: /usr/local/opt/lzo/include/lzo/ or with the brew command:
$ ls $(brew --prefix lzo)/include/lzo/
I have then tried to set the CFLAGS envronment variables, but still have the same error:
$ export EXTRA_CFLAGS=-I/usr/local/opt/lzo/include && export EXTRA_CXXFLAGS=-I/usr/local/opt/lzo/include && export EXTRA_LDFLAGS=-L/usr/local/opt/lzo/lib && pip install
And in desperation:
$ export EXTRA_CFLAGS=-I/usr/local/opt/lzo/include/lzo && export EXTRA_CXXFLAGS=-I/usr/local/opt/lzo/include/lzo && export EXTRA_LDFLAGS=-L/usr/local/opt/lzo/lib && pip install python-lzo
Still the same error message, any ideas?
— Updated–
Also tried passing the variables directly into pip:
pip install --global-option='build_ext' --global-option='-I/usr/local/opt' --global-option='-I/usr/local/opt/lzo/include/lzo' --global-option='-L/usr/local/opt/lzo/lib' python-lzo
This time I get a slightly different error:
cmdoptions.check_install_build_global(options)
Collecting python-lzo
Using cached python-lzo-1.11.tar.gz
Skipping bdist_wheel for python-lzo, due to binaries being disabled for it.
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... error
Complete output from command /Users/myuser/dir/devel/lzo_example/env/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-14vN3V/python-lzo/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" build_ext -I/usr/local/opt -I/usr/local/opt/lzo/include/lzo -L/usr/local/opt/lzo/lib install --record /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-Gq5pjs-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/emson/Dropbox/devel/aws/coursenut_bigdata/lzo_example/env/bin/../include/site/python2.7/python-lzo:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/opt/lzo/include/lzo -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
In file included from lzomodule.c:35:
/usr/local/opt/lzo/include/lzo/lzo1x.h:33:10: fatal error: 'lzo/lzoconf.h' file not found
#include <lzo/lzoconf.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
Many thanks
— SUCCESS —
Big thanks @daphtdazz your solution (see below) worked!
I have modified it slightly to use the brew --cellar command to get the location of the lzo files:
export C_INCLUDE_PATH=$(brew --cellar lzo)/2.09/include/lzo:$(brew --cellar lzo)/2.09/include/
export LIBRARY_PATH=/usr/local/lib
pip install python-lzo
Answers:
I don’t use MacOS, but can you try giving the env. variables directly to PIP?
So, same commands as you did in the second try (without exports), but without the &&.
foo=bar pim=pam pip install ...
Also, have you checked that the missing file exists in the directories you include?
I got the same results as you, and eventually this worked for me:
export C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.09/include/lzo:/usr/local/Cellar/lzo/2.09/include
export LIBRARY_PATH=/usr/local/lib
pip install python-lzo
(Obviously you may need to tweak those depending on what version of the lzo brew distribution you have installed.)
Which answered the question for me, but I don’t really understand why those aren’t configured correctly…
I have the Xcode developer tools installed though as well, and I think they’ve conflicted with brew and pip before, so maybe that’s it. I’ll keep having a look and if I work it out I’ll update this answer.
Edit
Having a look at the setup.py file for python-lzo it seems there’s a bit of weirdness going on, in particular I’m concerned that on Windows both include paths are being added, which suggests the lzo source distribution isn’t very well arranged. But there doesn’t seem to be a designed way of achieving this on macOS with a brew install, so I think the include part of it at least is best solved with the extra environment variable.
Excerpted from here, on M1 chip
# Install lzo homebrew
arch -arm64 brew install lzo
# Configure include path and lib path
export C_INCLUDE_PATH=/opt/homebrew/Cellar/lzo/2.10/include/lzo:/opt/homebrew/Cellar/lzo/2.10/include/
export LIBRARY_PATH=/opt/homebrew/lib
# Install
pip3 install python-lzo
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... done
Successfully installed python-lzo-1.12
Here is my log,
soner@Soners-MacBook-Pro ~ % arch -arm64 brew install lzo
==> Downloading https://ghcr.io/v2/homebrew/core/lzo/manifests/2.10
Already downloaded: /Users/soner/Library/Caches/Homebrew/downloads/d4aa5b0c239912c53bc857d1012c6b7feb4acb509618f5e100f95bf8521f08e7--lzo-2.10.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/lzo/blobs/sha256:e16072e8ef7a88
Already downloaded: /Users/soner/Library/Caches/Homebrew/downloads/2e2528259c080227ff9fcce6ab15d0f0415a85dad7f909d9a803d1f863f43960--lzo--2.10.arm64_monterey.bottle.tar.gz
==> Pouring lzo--2.10.arm64_monterey.bottle.tar.gz
/opt/homebrew/Cellar/lzo/2.10: 31 files, 565.6KB
==> Running `brew cleanup lzo`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
soner@Soners-MacBook-Pro ~ % export C_INCLUDE_PATH=/opt/homebrew/Cellar/lzo/2.10/include/lzo:/opt/homebrew/Cellar/lzo/2.10/include/
soner@Soners-MacBook-Pro ~ % export LIBRARY_PATH=/opt/homebrew/lib
soner@Soners-MacBook-Pro ~ % pip3 install python-lzo
Collecting python-lzo
Using cached python-lzo-1.14.tar.gz (14 kB)
Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for python-lzo, since package 'wheel' is not installed.
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... done
Successfully installed python-lzo-1.14
I’m trying to install python-lzo on mac OS Sierra. I have used brew to install the c lzo library:
brew update
brew install lzo
I then try and install python-lzo using a virtual environment and pip:
$ virtualenv ./env
$ . ./env/bin/activate
$ pip install python-lzo
Which gives me the following output:
Collecting python-lzo
Using cached python-lzo-1.11.tar.gz
Building wheels for collected packages: python-lzo
Running setup.py bdist_wheel for python-lzo: started
Running setup.py bdist_wheel for python-lzo: finished with status 'error'
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-wDRYjx/python-lzo/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" bdist_wheel -d /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/tmp_ueOY4pip-wheel- --python-tag cp27:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running bdist_wheel
running build
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
lzomodule.c:35:10: fatal error: 'lzo1x.h' file not found
#include <lzo1x.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
Running setup.py clean for python-lzo
Failed to build python-lzo
Installing collected packages: python-lzo
Running setup.py install for python-lzo: started
Running setup.py install for python-lzo: finished with status 'error'
Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-wDRYjx/python-lzo/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" install --record /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-8RUWxo-record/install-record.txt --single-version-externally-managed --compile:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running install
running build
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
lzomodule.c:35:10: fatal error: 'lzo1x.h' file not found
#include <lzo1x.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
The lzo1x.h file does exist here: /usr/local/opt/lzo/include/lzo/ or with the brew command:
$ ls $(brew --prefix lzo)/include/lzo/
I have then tried to set the CFLAGS envronment variables, but still have the same error:
$ export EXTRA_CFLAGS=-I/usr/local/opt/lzo/include && export EXTRA_CXXFLAGS=-I/usr/local/opt/lzo/include && export EXTRA_LDFLAGS=-L/usr/local/opt/lzo/lib && pip install
And in desperation:
$ export EXTRA_CFLAGS=-I/usr/local/opt/lzo/include/lzo && export EXTRA_CXXFLAGS=-I/usr/local/opt/lzo/include/lzo && export EXTRA_LDFLAGS=-L/usr/local/opt/lzo/lib && pip install python-lzo
Still the same error message, any ideas?
— Updated–
Also tried passing the variables directly into pip:
pip install --global-option='build_ext' --global-option='-I/usr/local/opt' --global-option='-I/usr/local/opt/lzo/include/lzo' --global-option='-L/usr/local/opt/lzo/lib' python-lzo
This time I get a slightly different error:
cmdoptions.check_install_build_global(options)
Collecting python-lzo
Using cached python-lzo-1.11.tar.gz
Skipping bdist_wheel for python-lzo, due to binaries being disabled for it.
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... error
Complete output from command /Users/myuser/dir/devel/lzo_example/env/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-build-14vN3V/python-lzo/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('rn', 'n');f.close();exec(compile(code, __file__, 'exec'))" build_ext -I/usr/local/opt -I/usr/local/opt/lzo/include/lzo -L/usr/local/opt/lzo/lib install --record /var/folders/38/v_r4bjvx7dz626x16nh6jcn00000gn/T/pip-Gq5pjs-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/emson/Dropbox/devel/aws/coursenut_bigdata/lzo_example/env/bin/../include/site/python2.7/python-lzo:
/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:251: UserWarning: 'licence' distribution option is deprecated; use 'license'
warnings.warn(msg)
running build_ext
building 'lzo' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/lzo -I/usr/local/opt/lzo/include/lzo -I/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lzomodule.c -o build/temp.macosx-10.11-x86_64-2.7/lzomodule.o
In file included from lzomodule.c:35:
/usr/local/opt/lzo/include/lzo/lzo1x.h:33:10: fatal error: 'lzo/lzoconf.h' file not found
#include <lzo/lzoconf.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
Many thanks
— SUCCESS —
Big thanks @daphtdazz your solution (see below) worked!
I have modified it slightly to use the brew --cellar command to get the location of the lzo files:
export C_INCLUDE_PATH=$(brew --cellar lzo)/2.09/include/lzo:$(brew --cellar lzo)/2.09/include/
export LIBRARY_PATH=/usr/local/lib
pip install python-lzo
I don’t use MacOS, but can you try giving the env. variables directly to PIP?
So, same commands as you did in the second try (without exports), but without the &&.
foo=bar pim=pam pip install ...
Also, have you checked that the missing file exists in the directories you include?
I got the same results as you, and eventually this worked for me:
export C_INCLUDE_PATH=/usr/local/Cellar/lzo/2.09/include/lzo:/usr/local/Cellar/lzo/2.09/include
export LIBRARY_PATH=/usr/local/lib
pip install python-lzo
(Obviously you may need to tweak those depending on what version of the lzo brew distribution you have installed.)
Which answered the question for me, but I don’t really understand why those aren’t configured correctly…
I have the Xcode developer tools installed though as well, and I think they’ve conflicted with brew and pip before, so maybe that’s it. I’ll keep having a look and if I work it out I’ll update this answer.
Edit
Having a look at the setup.py file for python-lzo it seems there’s a bit of weirdness going on, in particular I’m concerned that on Windows both include paths are being added, which suggests the lzo source distribution isn’t very well arranged. But there doesn’t seem to be a designed way of achieving this on macOS with a brew install, so I think the include part of it at least is best solved with the extra environment variable.
Excerpted from here, on M1 chip
# Install lzo homebrew
arch -arm64 brew install lzo
# Configure include path and lib path
export C_INCLUDE_PATH=/opt/homebrew/Cellar/lzo/2.10/include/lzo:/opt/homebrew/Cellar/lzo/2.10/include/
export LIBRARY_PATH=/opt/homebrew/lib
# Install
pip3 install python-lzo
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... done
Successfully installed python-lzo-1.12
Here is my log,
soner@Soners-MacBook-Pro ~ % arch -arm64 brew install lzo
==> Downloading https://ghcr.io/v2/homebrew/core/lzo/manifests/2.10
Already downloaded: /Users/soner/Library/Caches/Homebrew/downloads/d4aa5b0c239912c53bc857d1012c6b7feb4acb509618f5e100f95bf8521f08e7--lzo-2.10.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/lzo/blobs/sha256:e16072e8ef7a88
Already downloaded: /Users/soner/Library/Caches/Homebrew/downloads/2e2528259c080227ff9fcce6ab15d0f0415a85dad7f909d9a803d1f863f43960--lzo--2.10.arm64_monterey.bottle.tar.gz
==> Pouring lzo--2.10.arm64_monterey.bottle.tar.gz
/opt/homebrew/Cellar/lzo/2.10: 31 files, 565.6KB
==> Running `brew cleanup lzo`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
soner@Soners-MacBook-Pro ~ % export C_INCLUDE_PATH=/opt/homebrew/Cellar/lzo/2.10/include/lzo:/opt/homebrew/Cellar/lzo/2.10/include/
soner@Soners-MacBook-Pro ~ % export LIBRARY_PATH=/opt/homebrew/lib
soner@Soners-MacBook-Pro ~ % pip3 install python-lzo
Collecting python-lzo
Using cached python-lzo-1.14.tar.gz (14 kB)
Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for python-lzo, since package 'wheel' is not installed.
Installing collected packages: python-lzo
Running setup.py install for python-lzo ... done
Successfully installed python-lzo-1.14