Changing LD_LIBRARY_PATH at runtime for ctypes
Question:
How do you update this environment variable at runtime so that ctypes can load a library wherever? I’ve tried the following and neither seem to work.
from ctypes import *
os.environ['LD_LIBRARY_PATH'] = "/home/starlon/Projects/pyCFA635/lib"
os.putenv('LD_LIBRARY_PATH', "/home/starlon/Projects/pyCFA635/lib")
lib = CDLL("libevaluator.so")
Answers:
By the time a program such as Python is running, the dynamic loader (ld.so.1 or something similar) has already read LD_LIBRARY_PATH and won’t notice any changes thereafter. So, unless the Python software itself evaluates LD_LIBRARY_PATH and uses it to build the possible path name of the library for dlopen() or an equivalent function to use, setting the variable in the script will have no effect.
Given that you say it doesn’t work, it seems plausible to suppose that Python does not build and try all the possible library names; it probably relies on LD_LIBRARY_PATH alone.
CDLL can be passed a fully qualified path name, so for example I am using the following in one of my scripts where the .so is in the same directory as the python script.
import os
path = os.path.dirname(os.path.realpath(__file__))
dll = CDLL("%s/iface.so"%path)
In your case the following should suffice.
from ctypes import *
lib = CDLL("/home/starlon/Projects/pyCFA635/lib/libevaluator.so")
Even if you give a fully qualified path to CDLL or cdll.LoadLibrary(), you may still need to set LD_LIBRARY_PATH before invoking Python. If the shared library you load explicitly refers to another shared library and no “rpath” is set in the .so for that library, then it won’t be found, even if it has already been loaded. An rpath in a library specifies a search path to be used to search for other libraries needed by that library
For example, I have a case of a set of interdependent third-party libraries not produced by me. b.so references a.so. Even if I load a.so in advance:
ctypes.cdll.LoadLibrary('/abs/path/to/a.so')
ctypes.cdll.LoadLibrary('/abs/path/to/b.so')
I get an error on the second load, because b.so refers to simply ‘a.so’, without an rpath, and so b.so doesn’t know that’s the correct a.so. So I have to set LD_LIBRARY_PATH in advance to include ‘/abs/path/to’.
To avoid having to set LD_LIBRARY_PATH, you modify the rpath entry in the .so files. On Linux, there are two utilities I found that do this: chrpath, and patchelf. chrpath is available from the Ubuntu repositories. It cannot change rpath on .so’s that never had one. patchelf is more flexible.
Compile your binary with a rpath relative to the current working directory like:
gcc -shared -o yourbinary.so yoursource.c otherbinary.so
-Wl,-rpath='.',-rpath='./another/relative/rpath' -fpic
Then, you are able to change the working directory in python at runtime with:
import os
os.chdir('/path/to/your/binaries')
Like this, the loader also finds other dynamic libraries like otherbinary.so
Setting LD_LIBRARY_PATH to the path where libraries are placed won’t work here and ctypes will not notice any changes. So, you need to set this at the source and run ldconfig before your script to take this into account. Moreover setting the os environment or any PATH variable in the script will have no effect.
I was facing a similar issue and spent around a day to figure this out.
mkdir -p /etc/ld.so.conf.d/
echo "/home/starlon/Projects/pyCFA635/lib" > /etc/ld.so.conf.d/mycustomPath.conf
ldconfig
Then verify if the path is set with:
ldconfig -v | less
After this is done, try to run your script. This has worked for me and should work for you also.
You can see below URL which helped me to resolve this:
https://www.cyberciti.biz/faq/linux-setting-changing-library-path/
Note: I realized the question is old, however I wanted to contribute to this as the accepted answer alone was not actually solving my problem.
How do you update this environment variable at runtime so that ctypes can load a library wherever? I’ve tried the following and neither seem to work.
from ctypes import *
os.environ['LD_LIBRARY_PATH'] = "/home/starlon/Projects/pyCFA635/lib"
os.putenv('LD_LIBRARY_PATH', "/home/starlon/Projects/pyCFA635/lib")
lib = CDLL("libevaluator.so")
By the time a program such as Python is running, the dynamic loader (ld.so.1 or something similar) has already read LD_LIBRARY_PATH and won’t notice any changes thereafter. So, unless the Python software itself evaluates LD_LIBRARY_PATH and uses it to build the possible path name of the library for dlopen() or an equivalent function to use, setting the variable in the script will have no effect.
Given that you say it doesn’t work, it seems plausible to suppose that Python does not build and try all the possible library names; it probably relies on LD_LIBRARY_PATH alone.
CDLL can be passed a fully qualified path name, so for example I am using the following in one of my scripts where the .so is in the same directory as the python script.
import os
path = os.path.dirname(os.path.realpath(__file__))
dll = CDLL("%s/iface.so"%path)
In your case the following should suffice.
from ctypes import *
lib = CDLL("/home/starlon/Projects/pyCFA635/lib/libevaluator.so")
Even if you give a fully qualified path to CDLL or cdll.LoadLibrary(), you may still need to set LD_LIBRARY_PATH before invoking Python. If the shared library you load explicitly refers to another shared library and no “rpath” is set in the .so for that library, then it won’t be found, even if it has already been loaded. An rpath in a library specifies a search path to be used to search for other libraries needed by that library
For example, I have a case of a set of interdependent third-party libraries not produced by me. b.so references a.so. Even if I load a.so in advance:
ctypes.cdll.LoadLibrary('/abs/path/to/a.so')
ctypes.cdll.LoadLibrary('/abs/path/to/b.so')
I get an error on the second load, because b.so refers to simply ‘a.so’, without an rpath, and so b.so doesn’t know that’s the correct a.so. So I have to set LD_LIBRARY_PATH in advance to include ‘/abs/path/to’.
To avoid having to set LD_LIBRARY_PATH, you modify the rpath entry in the .so files. On Linux, there are two utilities I found that do this: chrpath, and patchelf. chrpath is available from the Ubuntu repositories. It cannot change rpath on .so’s that never had one. patchelf is more flexible.
Compile your binary with a rpath relative to the current working directory like:
gcc -shared -o yourbinary.so yoursource.c otherbinary.so
-Wl,-rpath='.',-rpath='./another/relative/rpath' -fpic
Then, you are able to change the working directory in python at runtime with:
import os
os.chdir('/path/to/your/binaries')
Like this, the loader also finds other dynamic libraries like otherbinary.so
Setting LD_LIBRARY_PATH to the path where libraries are placed won’t work here and ctypes will not notice any changes. So, you need to set this at the source and run ldconfig before your script to take this into account. Moreover setting the os environment or any PATH variable in the script will have no effect.
I was facing a similar issue and spent around a day to figure this out.
mkdir -p /etc/ld.so.conf.d/
echo "/home/starlon/Projects/pyCFA635/lib" > /etc/ld.so.conf.d/mycustomPath.conf
ldconfig
Then verify if the path is set with:
ldconfig -v | less
After this is done, try to run your script. This has worked for me and should work for you also.
You can see below URL which helped me to resolve this:
https://www.cyberciti.biz/faq/linux-setting-changing-library-path/
Note: I realized the question is old, however I wanted to contribute to this as the accepted answer alone was not actually solving my problem.