Installing ssdeep package from PyPi on M1 Macbook
Question:
The Goal
Install ssdeep PyPi package on a M1 Macbook Pro.
The Problem
When I run pip install ssdeep I get 2 errors
The first error is caused because fuzzy.h cannot be found.
warnings.warn(
running egg_info
creating /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info
writing /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/dependency_links.txt
writing requirements to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/requires.txt
writing top-level names to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/top_level.txt
writing manifest file '/private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/SOURCES.txt'
src/ssdeep/__pycache__/_ssdeep_cffi_a28e5628x27adcb8d.c:266:14: fatal error: 'fuzzy.h' file not found
#include "fuzzy.h"
^~~~~~~~~
1 error generated.
Traceback (most recent call last):
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py", line 186, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py", line 1007, in spawn
spawn(cmd, dry_run=self.dry_run, **kwargs)
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
The second error has to do with setuptools.installer being deprecated. I’m not sure this is all that important though. I think resolving the first error would resolve this one as well.
/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
Attempted Solutions
Solution 1: Install SSDeep with Homebrew brew install ssdeep
Result: pip install ssdeep has the same error about fuzzy.h missing
Solution 2: Use the prepackaged version of SSDeep BUILD_LIB=1 pip install ssdeep
Result: The error about fuzzy.h goes away but the second error regarding setuptools.installer being deprecated remains.
References
- Compiling SSDeep and pydeep on MacOS X 10.9+ This was pretty out of date though.
- SSDeep Documentation
Answers:
ssdeep package at PyPI is a Python wrapper for ssdeep library written in C. So first you have to compile and install ssdeep, then other python-ssdeep requirements, then compile and install python-ssdeep.
I found a solution. Essentially what’s going on is that Homebrew installs ssdeep in a location that the ssdeep PyPi package is not expecting. You can point the PyPi package to the correct locations with the following steps.
1: Install ssdeep with homebrew brew install ssdeep
2: List homebrew directories for ssdeep brew ls ssdeep
This produces output like
/opt/homebrew/Cellar/ssdeep/2.14.1/bin/ssdeep
/opt/homebrew/Cellar/ssdeep/2.14.1/include/ (2 files)
/opt/homebrew/Cellar/ssdeep/2.14.1/lib/libfuzzy.2.dylib
/opt/homebrew/Cellar/ssdeep/2.14.1/lib/ (2 other files)
/opt/homebrew/Cellar/ssdeep/2.14.1/share/man/man1/ssdeep.1
3: Set the LDFLAGS environment variable to the path of the ssdeep lib directory from the output in step 2.
export LDFLAGS="-L/opt/homebrew/Cellar/ssdeep/2.14.1/lib"
4: Set the C_INCLUDE_PATH environment variable to the path of the ssdeep include directory from the output in step 2.
export C_INCLUDE_PATH=/opt/homebrew/Cellar/ssdeep/2.14.1/include
5: Install ssdeep from PyPi pip install ssdeep
@HopAlongPolly answer almost worked for me but i got an error which the root cause seems to be:
ld: warning: ignoring file /opt/homebrew/Cellar/ssdeep/2.14.1/lib/libfuzzy.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
to solve this i ran BUILD_LIB=1 pip install ssdeep
If your env is pretty new you will get the following errors:
/bin/sh: libtoolize: command not found
/bin/sh: automake: command not found
to solve this you need to run brew install libtool automake and then create the following a symlink somewhere in your path from libtoolize to the glibtoolize binary that was installed via brew (This is needed as build process looks for libtoolize but homebrew installs glibtoolize). Pretty sure there is a cleaner way to point to the correct binary but the symlink did the job 😉
In summary do the steps that @HopAlongPolly recommended then run
brew install libtool automake
ln -s /opt/homebrew/bin/glibtoolize /opt/homebrew/bin/libtoolize
BUILD_LIB=1 pip install ssdeep
The Goal
Install ssdeep PyPi package on a M1 Macbook Pro.
The Problem
When I run pip install ssdeep I get 2 errors
The first error is caused because fuzzy.h cannot be found.
warnings.warn(
running egg_info
creating /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info
writing /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/dependency_links.txt
writing requirements to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/requires.txt
writing top-level names to /private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/top_level.txt
writing manifest file '/private/var/folders/0f/4c82tsj50n10zqq89fndcslc0000gn/T/pip-pip-egg-info-ai0atrdv/ssdeep.egg-info/SOURCES.txt'
src/ssdeep/__pycache__/_ssdeep_cffi_a28e5628x27adcb8d.c:266:14: fatal error: 'fuzzy.h' file not found
#include "fuzzy.h"
^~~~~~~~~
1 error generated.
Traceback (most recent call last):
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py", line 186, in _compile
self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py", line 1007, in spawn
spawn(cmd, dry_run=self.dry_run, **kwargs)
File "/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
The second error has to do with setuptools.installer being deprecated. I’m not sure this is all that important though. I think resolving the first error would resolve this one as well.
/Users/user/proj/.venv/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
Attempted Solutions
Solution 1: Install SSDeep with Homebrew brew install ssdeep
Result: pip install ssdeep has the same error about fuzzy.h missing
Solution 2: Use the prepackaged version of SSDeep BUILD_LIB=1 pip install ssdeep
Result: The error about fuzzy.h goes away but the second error regarding setuptools.installer being deprecated remains.
References
- Compiling SSDeep and pydeep on MacOS X 10.9+ This was pretty out of date though.
- SSDeep Documentation
ssdeep package at PyPI is a Python wrapper for ssdeep library written in C. So first you have to compile and install ssdeep, then other python-ssdeep requirements, then compile and install python-ssdeep.
I found a solution. Essentially what’s going on is that Homebrew installs ssdeep in a location that the ssdeep PyPi package is not expecting. You can point the PyPi package to the correct locations with the following steps.
1: Install ssdeep with homebrew brew install ssdeep
2: List homebrew directories for ssdeep brew ls ssdeep
This produces output like
/opt/homebrew/Cellar/ssdeep/2.14.1/bin/ssdeep
/opt/homebrew/Cellar/ssdeep/2.14.1/include/ (2 files)
/opt/homebrew/Cellar/ssdeep/2.14.1/lib/libfuzzy.2.dylib
/opt/homebrew/Cellar/ssdeep/2.14.1/lib/ (2 other files)
/opt/homebrew/Cellar/ssdeep/2.14.1/share/man/man1/ssdeep.1
3: Set the LDFLAGS environment variable to the path of the ssdeep lib directory from the output in step 2.
export LDFLAGS="-L/opt/homebrew/Cellar/ssdeep/2.14.1/lib"
4: Set the C_INCLUDE_PATH environment variable to the path of the ssdeep include directory from the output in step 2.
export C_INCLUDE_PATH=/opt/homebrew/Cellar/ssdeep/2.14.1/include
5: Install ssdeep from PyPi pip install ssdeep
@HopAlongPolly answer almost worked for me but i got an error which the root cause seems to be:
ld: warning: ignoring file /opt/homebrew/Cellar/ssdeep/2.14.1/lib/libfuzzy.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
to solve this i ran BUILD_LIB=1 pip install ssdeep
If your env is pretty new you will get the following errors:
/bin/sh: libtoolize: command not found
/bin/sh: automake: command not found
to solve this you need to run brew install libtool automake and then create the following a symlink somewhere in your path from libtoolize to the glibtoolize binary that was installed via brew (This is needed as build process looks for libtoolize but homebrew installs glibtoolize). Pretty sure there is a cleaner way to point to the correct binary but the symlink did the job 😉
In summary do the steps that @HopAlongPolly recommended then run
brew install libtool automake
ln -s /opt/homebrew/bin/glibtoolize /opt/homebrew/bin/libtoolize
BUILD_LIB=1 pip install ssdeep