Conan & rpaths

Dealing with dynamic linker errors in Conan

Kyle Downey

The last post introduced Conan for C/C++ library management. When I sat down to set up a CLion project using the LibUV and LibWebsockets dependencies though I hit an interesting quirk of MacOS X: the problems that its strict @rpath handling creates for package managers like Conan.

The problem

I set up a new project to create a client for the GDAX cryptocurrency exchange's websockets marketdata API. Since this was a simple case of declaring dependencies without any extra build required beyond CMake I used a conanfile.txt instead of Python, but either format would have worked. I declared a dependency on LWS plus fmtlib and GTest, setting up static linking for gtest & fmt and the CMake build generator:

[requires]
LibWebsockets/2.1@cloudwall/stable
fmt/3.0.0@memsharded/testing
gtest/1.8.0@lasote/stable

[options]
gtest:shared=False
fmt:shared=False

[generators]
cmake

CMakeLists.txt is equally simple, following the Conan online documentation:

cmake_minimum_required(VERSION 3.4)
project(libgdax)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14")

if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/conanbuildinfo.cmake)
    # Clion, with conanbuildinfo.cmake in root
    include(${CMAKE_CURRENT_SOURCE_DIR}/conanbuildinfo.cmake)
else()
    # regular CMake
    include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
endif()
conan_basic_setup()

include_directories("include")

add_library(gdax include/gdax.h src/gdaxclient-ws.cpp)
target_link_libraries(gdax ${CONAN_LIBS})

add_executable(gdaxclient-ws_test test/gdaxclient-ws_test.cpp)
target_link_libraries(gdaxclient-ws_test gdax ${CONAN_LIBS})

I set to work on the code expecting that most of the issues would come from the fact that it had been about 19 years since I last coded seriously in C++ and a lot had changed. Not only is the risk model that I was wrapping in CORBA services long ago replaced by SecDB and Slang, but with C++11, C++14 and now C++17 the language itself has changed as well. Plus a lot of brain cells in key locations had clearly died since then. A LOT. Like: the brain cells which remembered how to write a proper constructor, C++'s 101 meanings of const, and a lot else.

If you have a choice in your midlife crisis, I recommend a motorcycle.

OK, so wonderful, having spent a mortifyingly long time writing a trivial class it was time to run the first GTest and see if it worked.

Silvrback blog image

So what happened here?

LD_LIBRARY_PATH and @rpath

MacOS X is the fourth UNIX flavor that I have worked on as a programmer, and before this weekend I had never written C++ on it. So I was carrying around some assumptions about how dynamic library loading is supposed to work that proved to be hazardous.

On Linux and Solaris libraries and binaries are compiled with an rpath: a runtime path which can provide a default location for loading dynamic libraries or, depending on how you build it, inhibit loading the dynamic library from any other location. The "norm" that I am used to is the former behavior -- where LD_LIBRARY_PATH acts as a fallback -- but the latter is how MacOS X works. It forces you to load from the rpath cooked into the binary, or you have to rewrite it.

A partial fix

Going back to the Conan documentation to look for what they had to say about rpath handling, I found Pro Tip: Shared Libraries & rpaths.

TL;DR: set rpath to the library name only, with no path, and then set up your build to copy the dylibs alongside the binary. I added this to the conanfile.txt:

[imports]
bin, *.dll -> ./bin # Copies all dll files from packages bin folder to my "bin" folder
lib, *.dylib* -> ./bin # Copies all dylib files from packages lib folder to my "bin" folder

... aaaaaaand ... still no good.

LibUV issues

The above sorted out the LWS dependency, but the transitive dependency on LibUV still had problems, and the error made even less sense:

$ ./gdaxclient-ws_test
dyld: Library not loaded: /usr/local/lib/libuv.1.dylib
  Referenced from: /Users/kdowney/dev/shadow/libgdax/cmake-build-debug/bin/./gdaxclient-ws_test
  Reason: image not found
Abort trap: 6

Nowhere in the system, build files, etc. was there a reference to /usr/local/lib for LibUV, and the dylibs were in the right place alongside the binary. A check with otool confirmed the problem lay in the rpath setting for this dependency, but it was not clear at first where this came from.

This is one of the hazards of multiple levels of indirection in builds. In the case of the LibUV build you have Conan generating a configure script which in turn generates Makefiles. For LibWebsockets it is using Conan's CMake generator, which helpfully includes settings to set rpath to the library name only. All the build script generation lets you run cross-platform builds and not worry about these details ... until one of those details bites you.

The full fix

MacOS X's install_name_tool lets you fix this in a binary so you can effectively "relocate" a library from its intended install location in /usr/local/lib and allow it to work in a different directory:

install_name_tool -change /usr/local/lib/libuv.1.dylib libuv.1.dylib <path>

That will work, but ideally we want to build the dylib with the right path to start. A hunt on Google turned up at least one package already uploaded to Conan which took a cleaner approach by modifying the configure script to change the -install_name parameter in the Conan build method:

        self.run("chmod +x libuv-v1.9.1/autogen.sh")
        self.run("cd libuv-v1.9.1 && ./autogen.sh")

        # workaround for rpath
        if self.settings.os == "Macos":
            old_str = '-install_name \$rpath/\$soname'
            new_str = '-install_name \$soname'
            replace_in_file("./libuv-v1.9.1/configure", old_str, new_str)

After the autogen.sh script and before running configure you go in and edit the configure script to set -install_name to just $soname -- the dylib name in this case.

You can use otool to confirm the fix:

$ otool -L /Users/kdowney/.conan/data/LibUV/1.9.1/cloudwall/stable/package/a47fd1f3db1f83c7cb0da8cea82a3e1683ae91ea/lib/libuv.dylib
/Users/kdowney/.conan/data/LibUV/1.9.1/cloudwall/stable/package/a47fd1f3db1f83c7cb0da8cea82a3e1683ae91ea/lib/libuv.dylib:
    libuv.1.dylib (compatibility version 2.0.0, current version 2.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

Now the recommended Conan workaround can be effective: by copying libuv.1.dylib into the same directory as the binary as above the references will point to the right location.

Silvrback blog image