Lightgbm OSError, Library not loaded
Question:
If I simply do:
import lightgbm as lgb
I’m getting
python script.py
Traceback (most recent call last):
File "script.py", line 4, in <module>
import lightgbm as lgb
File "/usr/local/lib/python2.7/site-packages/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset
File "/usr/local/lib/python2.7/site-packages/lightgbm/basic.py", line 31, in <module>
_LIB = _load_lib()
File "/usr/local/lib/python2.7/site-packages/lightgbm/basic.py", line 26, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 440, in LoadLibrary
return self._dlltype(name)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib
Referenced from: /usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
I seem to have everything installed correct:
python -m pip install lightgbm
Requirement already satisfied: lightgbm in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: wheel in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: scikit-learn in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: scipy in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: numpy in /usr/local/lib/python2.7/site-packages (from lightgbm)
I do have a 6 folder in my GCC folder. Should I need to install another version of GCC for this to work?
Answers:
I find a similar problem here LightGBM
The answer and comment may help you.
Build LightGBM in Mac:
brew install cmake
brew install gcc --without-multilib
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
mkdir build ; cd build
cmake ..
make -j
Then install:
cd ../python-packages
sudo python setup.py install --precompile
As stated by @ecodan, you might need to force Mac to use GCC and G++ instead of the default compiler. So instead of building with cmake .., try:
cmake -DCMAKE_C_COMPILER=/usr/local/Cellar/gcc/6.1.0/bin/gcc-6 -DCMAKE_CXX_COMPILER=/usr/local/Cellar/gcc/6.1.0/bin/g++-6 ..
ajusting the versions to match yours.
For users with macports, replace the beginning of the brew solution with:
port install gcc7 cmakeexport CXX=g++-mp-7 CC=gcc-mp-7
On MacOS High Sierra with MacPorts installed, I did the following:
- Install
clang-5.0 using MacPorts
- Inside the
/build directory, run
cmake -DCMAKE_CXX_COMPILER=clang++-mp-5.0 -DCMAKE_C_COMPILER=clang-mp-5.0 ..
- To build the python package, go to
/python_package directory and modify the setup.py script.
You need to modify the function compile_cpp() at the very end that checks the case for other OS (including Mac). Before the silent_call(...), add the following two lines:
cmake_cmd.append("-DCMAKE_CXX_COMPILER=clang++-mp-5.0")
cmake_cmd.append("-DCMAKE_C_COMPILER=clang-mp-5.0")
- Run
sudo python setup.py install. Enjoy
On MAC you need to install open-mpi:
brew install open-mpi
Same error, different source: seems like I had the gcc 8 version installed, and it needs gcc 7.
It worked by switching it back to last gcc 7 version:
brew switch gcc 7.3.0_1
All the above answers didn’t work for me.
On Mac, if I installed the libomp using brew fixed the problem: Refer: link
brew install libomp
I used this command generated from PyCharm and it worked for me.
conda install -p { < =replace with USER_HOME_DIR>}/anaconda3 lightgbm -y
Note : i tried Brew and pip install but they didn’t work for me as Azure ML studio already downloaded other versions of LightGBM that were causing conflict in upgrading to correct packages.
I had the same exact problem on M1 MAC. I have tried to import it through Jupiter notebook. This command solved the problem:
conda install lightgbm
conda install -c conda-forge lightgbm
If I simply do:
import lightgbm as lgb
I’m getting
python script.py
Traceback (most recent call last):
File "script.py", line 4, in <module>
import lightgbm as lgb
File "/usr/local/lib/python2.7/site-packages/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset
File "/usr/local/lib/python2.7/site-packages/lightgbm/basic.py", line 31, in <module>
_LIB = _load_lib()
File "/usr/local/lib/python2.7/site-packages/lightgbm/basic.py", line 26, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 440, in LoadLibrary
return self._dlltype(name)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib
Referenced from: /usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
I seem to have everything installed correct:
python -m pip install lightgbm
Requirement already satisfied: lightgbm in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: wheel in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: scikit-learn in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: scipy in /usr/local/lib/python2.7/site-packages (from lightgbm)
Requirement already satisfied: numpy in /usr/local/lib/python2.7/site-packages (from lightgbm)
I do have a 6 folder in my GCC folder. Should I need to install another version of GCC for this to work?
I find a similar problem here LightGBM
The answer and comment may help you.
Build LightGBM in Mac:
brew install cmake
brew install gcc --without-multilib
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
mkdir build ; cd build
cmake ..
make -j
Then install:
cd ../python-packages
sudo python setup.py install --precompile
As stated by @ecodan, you might need to force Mac to use GCC and G++ instead of the default compiler. So instead of building with cmake .., try:
cmake -DCMAKE_C_COMPILER=/usr/local/Cellar/gcc/6.1.0/bin/gcc-6 -DCMAKE_CXX_COMPILER=/usr/local/Cellar/gcc/6.1.0/bin/g++-6 ..
ajusting the versions to match yours.
For users with macports, replace the beginning of the brew solution with:
port install gcc7 cmakeexport CXX=g++-mp-7 CC=gcc-mp-7
On MacOS High Sierra with MacPorts installed, I did the following:
- Install
clang-5.0using MacPorts - Inside the
/builddirectory, run
cmake -DCMAKE_CXX_COMPILER=clang++-mp-5.0 -DCMAKE_C_COMPILER=clang-mp-5.0 .. - To build the python package, go to
/python_packagedirectory and modify thesetup.pyscript.
You need to modify the functioncompile_cpp()at the very end that checks the case for other OS (including Mac). Before thesilent_call(...), add the following two lines:
cmake_cmd.append("-DCMAKE_CXX_COMPILER=clang++-mp-5.0")
cmake_cmd.append("-DCMAKE_C_COMPILER=clang-mp-5.0")
- Run
sudo python setup.py install. Enjoy
On MAC you need to install open-mpi:
brew install open-mpi
Same error, different source: seems like I had the gcc 8 version installed, and it needs gcc 7.
It worked by switching it back to last gcc 7 version:
brew switch gcc 7.3.0_1
All the above answers didn’t work for me.
On Mac, if I installed the libomp using brew fixed the problem: Refer: link
brew install libomp
I used this command generated from PyCharm and it worked for me.
conda install -p { < =replace with USER_HOME_DIR>}/anaconda3 lightgbm -y
Note : i tried Brew and pip install but they didn’t work for me as Azure ML studio already downloaded other versions of LightGBM that were causing conflict in upgrading to correct packages.
I had the same exact problem on M1 MAC. I have tried to import it through Jupiter notebook. This command solved the problem:
conda install lightgbm
conda install -c conda-forge lightgbm