Gevent cant be installed on M1 mac using poetry
Question:
I tried to install many dependencies for a virtual environment using poetry. When it gets to gevent (20.9.0) it gets the following
import error:
ImportError: dlopen(/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so,
0x0002): tried:
'/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so'
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_cffi_backend.cpython-38-darwin.so' (no such file), '/usr/lib/_cffi_backend.cpython-38-darwin.so' (no such file)
I’ve tried to use pip3 instead, but still had the same problem.
Answers:
The M1 needs either all x86 code or all arm64 code in a process. This issue is caused by a program attempting to load an x86_64-only library from a native arm64 process. This unfortunately can’t be done (See here).
With specific regard to gevent, the project page here includes the following information:
Beginning with gevent 20.12.0, 64-bit ARM binaries are distributed on PyPI for aarch64 manylinux2014 compatible systems. Installing these needs a very recent version of pip. These wheels do not contain the c-ares resolver, are not tested, and are built with very low levels of optimizations. Serious production users of gevent on 64-bit ARM systems are encouraged to build their own binary wheels.
I’m not sure if arm64 is exclusively supported on Linux, but in any case, I would suggest updating your versions of both gevent and pip to the lastest possible versions.
I’ve have this problem with other libraries also and this solution worked some times:
sudo arch -arm64 <poetry or pip> install <lib to istall>
Using arch -arm64 allowed me to install the rigt wheel for the M1 processor
You need to compile it from source.
https://www.gevent.org/development/installing_from_source.html
arch -arm64 pip install --no-binary gevent gevent
I had to install it on my pc (not in virtual env). Then installing inside the virtual env worked.
So on Apple M1, run
sudo arch -arm64 pip3 install --no-binary gevent gevent
and then create and activate your python virtual env,
python3 -m venv env
source env/bin/activate
pip install -U pip
pip install gevent
You are done if you see the output like:
Collecting gevent
Using cached gevent-22.10.2.tar.gz (6.6 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting zope.event
Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB)
Collecting greenlet>=2.0.0
Using cached greenlet-2.0.2.tar.gz (164 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: setuptools in ./env/lib/python3.9/site-packages (from gevent) (58.0.4)
Collecting zope.interface
Downloading zope.interface-6.0-cp39-cp39-macosx_11_0_arm64.whl (202 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 202.4/202.4 kB 2.9 MB/s eta 0:00:00
Building wheels for collected packages: gevent
Building wheel for gevent (pyproject.toml) ... done
Created wheel for gevent: filename=gevent-22.10.2-cp39-cp39-macosx_10_9_universal2.whl size=2908594 sha256=c7184d20566f559ddec3b60250fbf9b8ffda973c14c93f70fc8ae22aa7b85521
Stored in directory: /Users/pavankrn/Library/Caches/pip/wheels/7f/f7/e0/8b48eb5aabb2beca135a3fdde34df3b6938cfccf4def985a45
Successfully built gevent
Installing collected packages: zope.interface, zope.event, greenlet, gevent
DEPRECATION: greenlet is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for greenlet ... done
If you freeze your requirements pip freeze > requirements.txt you will find the following packages.
gevent==22.10.2
greenlet==2.0.2
zope.event==4.6
zope.interface==6.0
I tried to install many dependencies for a virtual environment using poetry. When it gets to gevent (20.9.0) it gets the following
import error:
ImportError: dlopen(/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so,
0x0002): tried:
'/private/var/folders/21/wxg5bdsj1w3f3j_9sl_pktbw0000gn/T/pip-build-env-50mwte36/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so'
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_cffi_backend.cpython-38-darwin.so' (no such file), '/usr/lib/_cffi_backend.cpython-38-darwin.so' (no such file)
I’ve tried to use pip3 instead, but still had the same problem.
The M1 needs either all x86 code or all arm64 code in a process. This issue is caused by a program attempting to load an x86_64-only library from a native arm64 process. This unfortunately can’t be done (See here).
With specific regard to gevent, the project page here includes the following information:
Beginning with gevent 20.12.0, 64-bit ARM binaries are distributed on PyPI for aarch64 manylinux2014 compatible systems. Installing these needs a very recent version of pip. These wheels do not contain the c-ares resolver, are not tested, and are built with very low levels of optimizations. Serious production users of gevent on 64-bit ARM systems are encouraged to build their own binary wheels.
I’m not sure if arm64 is exclusively supported on Linux, but in any case, I would suggest updating your versions of both gevent and pip to the lastest possible versions.
I’ve have this problem with other libraries also and this solution worked some times:
sudo arch -arm64 <poetry or pip> install <lib to istall>
Using arch -arm64 allowed me to install the rigt wheel for the M1 processor
You need to compile it from source.
https://www.gevent.org/development/installing_from_source.html
arch -arm64 pip install --no-binary gevent gevent
I had to install it on my pc (not in virtual env). Then installing inside the virtual env worked.
So on Apple M1, run
sudo arch -arm64 pip3 install --no-binary gevent gevent
and then create and activate your python virtual env,
python3 -m venv env
source env/bin/activate
pip install -U pip
pip install gevent
You are done if you see the output like:
Collecting gevent
Using cached gevent-22.10.2.tar.gz (6.6 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting zope.event
Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB)
Collecting greenlet>=2.0.0
Using cached greenlet-2.0.2.tar.gz (164 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: setuptools in ./env/lib/python3.9/site-packages (from gevent) (58.0.4)
Collecting zope.interface
Downloading zope.interface-6.0-cp39-cp39-macosx_11_0_arm64.whl (202 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 202.4/202.4 kB 2.9 MB/s eta 0:00:00
Building wheels for collected packages: gevent
Building wheel for gevent (pyproject.toml) ... done
Created wheel for gevent: filename=gevent-22.10.2-cp39-cp39-macosx_10_9_universal2.whl size=2908594 sha256=c7184d20566f559ddec3b60250fbf9b8ffda973c14c93f70fc8ae22aa7b85521
Stored in directory: /Users/pavankrn/Library/Caches/pip/wheels/7f/f7/e0/8b48eb5aabb2beca135a3fdde34df3b6938cfccf4def985a45
Successfully built gevent
Installing collected packages: zope.interface, zope.event, greenlet, gevent
DEPRECATION: greenlet is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for greenlet ... done
If you freeze your requirements pip freeze > requirements.txt you will find the following packages.
gevent==22.10.2
greenlet==2.0.2
zope.event==4.6
zope.interface==6.0