Occur "Could NOT find Arrow" error when using pip_pypy3 to install pyarrow
Question:
I am trying to use pypy3 to install pyarrow, but some errors occur.
Basic information is blow:
macOS 10.15.7
Xcode 12.3
python version 3.7.9
pypy3 version 7.3.3
pyarrow version 0.17.1
cmd is 'pip_pypy3 install pyarrow==0.17.1'
Some key information and error content in the log:
...
Requirement already satisfied: numpy>=1.14 in /usr/local/Cellar/pypy3/7.3.3/libexec/site-packages (from pyarrow==0.17.1) (1.19.5)
...
cmake -DPYTHON_EXECUTABLE=/usr/local/Cellar/pypy3/7.3.3/bin/pypy3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-install-jagh2frg/pyarrow_522bc325fbd74d9ebdf84f29e3a66c0c
...
2021-01-10T21:19:27,670 -- Found Python3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (found version "3.7.9") found components: Interpreter Development NumPy Development.Module Development.Embed
2021-01-10T21:19:27,713 -- Found Python3Alt: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
2021-01-10T21:19:28,226 CMake Warning (dev) at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
2021-01-10T21:19:28,227 The package name passed to `find_package_handle_standard_args` (PkgConfig)
2021-01-10T21:19:28,227 does not match the name of the calling package (Arrow). This can lead to
2021-01-10T21:19:28,227 problems in calling code that expects `find_package` result variables
2021-01-10T21:19:28,227 (e.g., `_FOUND`) to follow a certain pattern.
2021-01-10T21:19:28,227 Call Stack (most recent call first):
2021-01-10T21:19:28,227 /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPkgConfig.cmake:67 (find_package_handle_standard_args)
2021-01-10T21:19:28,228 cmake_modules/FindArrow.cmake:39 (include)
2021-01-10T21:19:28,228 cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,228 CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,228 This warning is for project developers. Use -Wno-dev to suppress it.
2021-01-10T21:19:28,229 -- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
2021-01-10T21:19:28,255 -- Could NOT find Arrow (missing: Arrow_DIR)
2021-01-10T21:19:28,255 -- Checking for module 'arrow'
2021-01-10T21:19:28,269 -- No package 'arrow' found
2021-01-10T21:19:28,270 CMake Error at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
2021-01-10T21:19:28,270 **Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR**
2021-01-10T21:19:28,270 **ARROW_FULL_SO_VERSION ARROW_SO_VERSION)**
2021-01-10T21:19:28,271 Call Stack (most recent call first):
2021-01-10T21:19:28,271 /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
2021-01-10T21:19:28,271 cmake_modules/FindArrow.cmake:412 (find_package_handle_standard_args)
2021-01-10T21:19:28,271 cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,271 CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,272 -- Configuring incomplete, errors occurred!
Do I need to add some environment variables(such as ARROW_INCLUDE_DIR) to complete the installation?
According to what Uwe L. Korn said, I have installed arrow c++ using brew install brew install apache-arrow and brew install apache-arrow-glib, but there is another problem.
2021-01-11T17:44:05,139 creating build/bdist.macosx-10.7-x86_64/wheel/pyarrow/include
2021-01-11T17:44:05,140 error: can't copy 'build/lib.macosx-10.7-x86_64-3.7/pyarrow/include/arrow': doesn't exist or not a regular file
2021-01-11T17:44:05,149 ERROR: Failed building wheel for pyarrow
2021-01-11T17:44:05,151 Failed to build pyarrow
2021-01-11T17:44:05,152 ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,152 Exception information:
2021-01-11T17:44:05,152 Traceback (most recent call last):
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/base_command.py", line 224, in _main
2021-01-11T17:44:05,152 status = self.run(options, args)
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/req_command.py", line 180, in wrapper
2021-01-11T17:44:05,152 return func(self, options, args)
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/commands/install.py", line 364, in run
2021-01-11T17:44:05,152 ", ".join(pep517_build_failure_names)
2021-01-11T17:44:05,152 pip._internal.exceptions.InstallationError: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,200 Removed build tracker: '/private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-req-tracker-lp8m3f4g'
Answers:
The answer to your compilation problem will be that you should install Arrow C++ first. This will though only bring you a single step further and you will hit a new roadblock next as pyarrow is currently not working with PyPy, only CPython. There is an open upstream issue ARROW-2651 where you can track the progress but currently it seems nobody is working on that. So in your case you can either stick to CPython or help in that issue to fix the missing bits to get it running on PyPy.
Install Arrow C++, see documentation C++ and GLib (C) Packages for Debian GNU/Linux, Ubuntu, AlmaLinux, CentOS and Amazon Linux at https://arrow.apache.org/install/
This is what worked for me.
see: https://apache.jfrog.io/artifactory/arrow/
and do something like this for your version:
wget https://apache.jfrog.io/artifactory/arrow/ubuntu/apache-arrow-apt-source-latest-jammy.deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-jammy.deb
sudo apt update
sudo apt install -y -V libarrow-glib-dev # For GLib (C)
sudo apt install -y -V libarrow-dev # For C++
Also, I downgraded from python 3.11.0rc2 to 3.10.8
And then re-ran pip install
The error came from trying to install some stuff for dbt:
snowflake-connector-python[secure-local-storage]<2.8.2,>=2.4.1
I am trying to use pypy3 to install pyarrow, but some errors occur.
Basic information is blow:
macOS 10.15.7
Xcode 12.3
python version 3.7.9
pypy3 version 7.3.3
pyarrow version 0.17.1
cmd is 'pip_pypy3 install pyarrow==0.17.1'
Some key information and error content in the log:
...
Requirement already satisfied: numpy>=1.14 in /usr/local/Cellar/pypy3/7.3.3/libexec/site-packages (from pyarrow==0.17.1) (1.19.5)
...
cmake -DPYTHON_EXECUTABLE=/usr/local/Cellar/pypy3/7.3.3/bin/pypy3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-install-jagh2frg/pyarrow_522bc325fbd74d9ebdf84f29e3a66c0c
...
2021-01-10T21:19:27,670 -- Found Python3: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (found version "3.7.9") found components: Interpreter Development NumPy Development.Module Development.Embed
2021-01-10T21:19:27,713 -- Found Python3Alt: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
2021-01-10T21:19:28,226 CMake Warning (dev) at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
2021-01-10T21:19:28,227 The package name passed to `find_package_handle_standard_args` (PkgConfig)
2021-01-10T21:19:28,227 does not match the name of the calling package (Arrow). This can lead to
2021-01-10T21:19:28,227 problems in calling code that expects `find_package` result variables
2021-01-10T21:19:28,227 (e.g., `_FOUND`) to follow a certain pattern.
2021-01-10T21:19:28,227 Call Stack (most recent call first):
2021-01-10T21:19:28,227 /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPkgConfig.cmake:67 (find_package_handle_standard_args)
2021-01-10T21:19:28,228 cmake_modules/FindArrow.cmake:39 (include)
2021-01-10T21:19:28,228 cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,228 CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,228 This warning is for project developers. Use -Wno-dev to suppress it.
2021-01-10T21:19:28,229 -- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
2021-01-10T21:19:28,255 -- Could NOT find Arrow (missing: Arrow_DIR)
2021-01-10T21:19:28,255 -- Checking for module 'arrow'
2021-01-10T21:19:28,269 -- No package 'arrow' found
2021-01-10T21:19:28,270 CMake Error at /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
2021-01-10T21:19:28,270 **Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR**
2021-01-10T21:19:28,270 **ARROW_FULL_SO_VERSION ARROW_SO_VERSION)**
2021-01-10T21:19:28,271 Call Stack (most recent call first):
2021-01-10T21:19:28,271 /usr/local/Cellar/cmake/3.19.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
2021-01-10T21:19:28,271 cmake_modules/FindArrow.cmake:412 (find_package_handle_standard_args)
2021-01-10T21:19:28,271 cmake_modules/FindArrowPython.cmake:46 (find_package)
2021-01-10T21:19:28,271 CMakeLists.txt:210 (find_package)
2021-01-10T21:19:28,272 -- Configuring incomplete, errors occurred!
Do I need to add some environment variables(such as ARROW_INCLUDE_DIR) to complete the installation?
According to what Uwe L. Korn said, I have installed arrow c++ using brew install brew install apache-arrow and brew install apache-arrow-glib, but there is another problem.
2021-01-11T17:44:05,139 creating build/bdist.macosx-10.7-x86_64/wheel/pyarrow/include
2021-01-11T17:44:05,140 error: can't copy 'build/lib.macosx-10.7-x86_64-3.7/pyarrow/include/arrow': doesn't exist or not a regular file
2021-01-11T17:44:05,149 ERROR: Failed building wheel for pyarrow
2021-01-11T17:44:05,151 Failed to build pyarrow
2021-01-11T17:44:05,152 ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,152 Exception information:
2021-01-11T17:44:05,152 Traceback (most recent call last):
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/base_command.py", line 224, in _main
2021-01-11T17:44:05,152 status = self.run(options, args)
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/cli/req_command.py", line 180, in wrapper
2021-01-11T17:44:05,152 return func(self, options, args)
2021-01-11T17:44:05,152 File "/usr/local/Cellar/pypy3/7.3.3/libexec/site-packages/pip-20.3.3-py3.7.egg/pip/_internal/commands/install.py", line 364, in run
2021-01-11T17:44:05,152 ", ".join(pep517_build_failure_names)
2021-01-11T17:44:05,152 pip._internal.exceptions.InstallationError: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
2021-01-11T17:44:05,200 Removed build tracker: '/private/var/folders/7p/d9yrtx8s2h94h9bh3x801zmr0000gn/T/pip-req-tracker-lp8m3f4g'
The answer to your compilation problem will be that you should install Arrow C++ first. This will though only bring you a single step further and you will hit a new roadblock next as pyarrow is currently not working with PyPy, only CPython. There is an open upstream issue ARROW-2651 where you can track the progress but currently it seems nobody is working on that. So in your case you can either stick to CPython or help in that issue to fix the missing bits to get it running on PyPy.
Install Arrow C++, see documentation C++ and GLib (C) Packages for Debian GNU/Linux, Ubuntu, AlmaLinux, CentOS and Amazon Linux at https://arrow.apache.org/install/
This is what worked for me.
see: https://apache.jfrog.io/artifactory/arrow/
and do something like this for your version:
wget https://apache.jfrog.io/artifactory/arrow/ubuntu/apache-arrow-apt-source-latest-jammy.deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-jammy.deb
sudo apt update
sudo apt install -y -V libarrow-glib-dev # For GLib (C)
sudo apt install -y -V libarrow-dev # For C++
Also, I downgraded from python 3.11.0rc2 to 3.10.8
And then re-ran pip install
The error came from trying to install some stuff for dbt:
snowflake-connector-python[secure-local-storage]<2.8.2,>=2.4.1