Dylib built on CI can't be loaded

Question:

I’m building a Rust binary (liblonlat_bng.dylib) on Travis CI, pulling it into a Cython extension (in the same dir as the Cython source .c/.pyx), and testing it, also on Travis CI (in a different repo and build). However, tests of the Python package are failing, and I’m not sure why:

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/travis/build/urschrei/convertbng/test/test_convertbng.py", line 15, in <module>
    from convertbng.cutil import convert_bng as cconvert_bng
ImportError: dlopen(/Users/travis/build/urschrei/convertbng/convertbng/cutil.so, 2): Library not loaded: /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib
  Referenced from: /Users/travis/build/urschrei/convertbng/convertbng/cutil.so
  Reason: image not found

Here’s the OSX build output for the package from Travis:

Installing collected packages: convertbng
  Running setup.py develop for convertbng
    Running command /Users/travis/build/urschrei/convertbng/venv/bin/python2.7 -c "import setuptools, tokenize;__file__='/Users/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" develop --no-deps
    running develop
    running egg_info
    writing requirements to convertbng.egg-info/requires.txt
    writing convertbng.egg-info/PKG-INFO
    writing top-level names to convertbng.egg-info/top_level.txt
    writing dependency_links to convertbng.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found
    reading manifest file 'convertbng.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'convertbng.egg-info/SOURCES.txt'
    running build_ext
    building 'convertbng.cutil' extension
    creating build
    creating build/temp.macosx-10.11-x86_64-2.7
    creating build/temp.macosx-10.11-x86_64-2.7/convertbng
    clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -Iconvertbng -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c convertbng/cutil.c -o build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -O3
    [unused function warnings]
    creating build/lib.macosx-10.11-x86_64-2.7
    creating build/lib.macosx-10.11-x86_64-2.7/convertbng
    clang -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so
    copying build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so -> convertbng
    Creating /Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/convertbng.egg-link (link to .)
    Adding convertbng 0.4.14 to easy-install.pth file
    Installed /Users/travis/build/urschrei/convertbng
Successfully installed convertbng

And here’s the Linux Travis output, which successfully locates the dylib on a relative path. Note the $ORIGIN argument to -R, which can’t be used on OSX:

Installing collected packages: convertbng
  Running setup.py develop for convertbng
    Running command /usr/bin/python -c "import setuptools, tokenize;__file__='/home/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('rn', 'n'), __file__, 'exec'))" develop --no-deps
    running develop
    running egg_info
    writing requirements to convertbng.egg-info/requires.txt
    writing convertbng.egg-info/PKG-INFO
    writing top-level names to convertbng.egg-info/top_level.txt
    writing dependency_links to convertbng.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found
    reading manifest file 'convertbng.egg-info/SOURCES.txt'
    writing manifest file 'convertbng.egg-info/SOURCES.txt'
    running build_ext
    building 'convertbng.cutil' extension
    creating build
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/convertbng
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iconvertbng -I/usr/include/python2.7 -c convertbng/cutil.c -o build/temp.linux-x86_64-2.7/convertbng/cutil.o -O3
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/convertbng
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -Wl,-R$ORIGIN -llonlat_bng -o build/lib.linux-x86_64-2.7/convertbng/cutil.so
    copying build/lib.linux-x86_64-2.7/convertbng/cutil.so -> convertbng
    Creating /usr/local/lib/python2.7/dist-packages/convertbng.egg-link (link to .)
    Adding convertbng 0.4.14 to easy-install.pth file
    Installed /home/travis/build/urschrei/convertbng
Successfully installed convertbng

Other details:

  • Both my OS X and Travis are using XCode 7.3
  • The binaries are being built with the same commit both locally and on Travis
  • If I build the binary locally, then run setup.py build_ext --inplace on my package, the tests pass
  • I run the same install for Linux, using a .so built with the same commit on Travis, and the tests pass
  • If I copy the Travis-built .dylib to my local machine, then run setup.py build_ext --inplace, the tests fail with the same error as on Travis.

I’m very puzzled as to what’s going on. Is there something I should be looking at in the binaries? The otool -l output differs slightly. For instance:

Local:

cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan  1 01:00:02 1970
current version 1226.10.1

Travis:

cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan  1 00:00:02 1970
current version 1225.1.1

Link to passing Linux job: https://travis-ci.org/urschrei/convertbng/jobs/136730347
Link to failing OSX job: https://travis-ci.org/urschrei/convertbng/jobs/136730348

UPDATE:
It’s definitely because of the library name. if I use install_name_tool to change the dylib location to @loader_path/liblonlat_bng.dylib:

install_name_tool -change /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib @loader_path/liblonlat_bng.dylib convertbng/cutil.so

The Travis-built executable will run on my local machine. However, Travis seems to have a broken install_name_tool install, and I’m not managing to pass the correct invocation to cutil.so from setup.py. I’ve tried setting extra_link_args to ['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib'], which generates

/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.6-intel-2.7/convertbng/cutil.so -Wl,-rpath,@loader_path/liblonlat_bng.dylib

But my tests still fail if I do that.

UPDATE 2: I can fix the install_name of the dylib in the link phase when building with cargo:

RUSTFLAGS="-C link-args=-Wl,-install_name,@rpath/liblonlat_bng.dylib" cargo build --release

And this works. But I suspect my setup.py Extension setup is wrong:

# only append the runtime dir on Linux
rdirs = []
ldirs = []
if sys.platform != 'darwin':
    # from http://stackoverflow.com/a/10252190/416626
    # the $ORIGIN trick is not perfect, though
    rdirs = ['$ORIGIN']
if sys.platform == 'darwin':
    ldirs = ['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib']

extensions = Extension("convertbng.cutil",
                    sources=["convertbng/cutil" + suffix],
                    libraries=["lonlat_bng"],
                    include_dirs=['.', 'convertbng'],
                    library_dirs=['.', 'convertbng'],
                    runtime_library_dirs=rdirs,
                    extra_compile_args=["-O3"],
                    extra_link_args=ldirs
)
Asked By: urschrei

||
Source

Answers:

There are two ways to fix this without using install_name_tool.
Assuming that your dylib is in the same directory as your Cython extension (In this case, cutil.so)

  • Compile your Rust binary with rpath support, and correctly set the rpath in your setup.py Extension instance:
  • Invoke RUSTFLAGS="-C rpath" cargo build --release.
  • The less flexible way is to compile your Rust binary and pass the path in via linker flags. This bakes in the path (albeit with dynamic refs), however:
    • cargo rustc --release -- -C link-args=-Wl,-install_name,@rpath/libname.dylib

After you’ve enabled rpath support in your Rust dylib, edit your setup.py:

Set (or add) extra_link_args=["-Wl,-rpath", "-Wl,@loader_path/"] in your Extension instance.

You’ll know it’s worked if you compile your Cython extension (e.g. using setup.py build_ext --inplace), then run otool -l on the resulting .so:

   @rpath/liblonlat_bng.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

If you want your Rust dylib elsewhere relative to your Cython extension, you’ll need to adjust your @rpath/ or @loader_path/ paths in setup.py accordingly.

Note: The foregoing works for relative paths on OS X only. On Linux, you can set extra_link_args=["-Wl,-rpath", "-Wl,$ORIGIN"] in your setup.py Extension instance (and adjust it accordingly if your Rust dylib is elsewhere, relative to your extension.)

Answered By: urschrei

I had the same problem and I believe it is a bug in Distutils. I’ve made a pull request to fix this and I’ve also proposed a workaround. See https://github.com/python/cpython/pull/12418

Can you try the following in your setup.py?

from Cython.Distutils.build_ext import new_build_ext as build_ext
# alternative:
# from distutils.command import build_ext

class my_build_ext(build_ext):
    """Workaround for rpath bug in distutils for OSX."""

    def finalize_options(self):
        super().finalize_options()
        # Special treatment of rpath in case of OSX, to work around python
        # distutils bug 36353. This constructs proper rpath arguments for clang.
        # See https://bugs.python.org/issue36353
        if sys.platform[:6] == "darwin":
            for path in self.rpath:
                for ext in self.extensions:
                    ext.extra_link_args.append("-Wl,-rpath," + path)
            self.rpath[:] = []

setup(
    cmdclass={'build_ext': my_build_ext}
    # ...
)
Answered By: Toon Verstraelen
Categories: questions Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.