Tests fail linking against libarrow.so built in Conda environment

[pitrou]

I have a Conda environment where I do Arrow development.
When I try to build nanoarrow in that same environment I get linking errors in tests.

[23/25] Linking CXX executable utils_test
FAILED: utils_test utils_test[1]_tests.cmake /home/antoine/arrow/nanoarrow/build-debug/utils_test[1]_tests.cmake 
: && /home/antoine/mambaforge/envs/pyarrow/bin/g++ -g  CMakeFiles/utils_test.dir/src/nanoarrow/utils_test.cc.o -o utils_test  -Wl,-rpath,/home/antoine/mambaforge/envs/pyarrow/lib  libnanoarrow.a  lib/libgtest_maind.a  /home/antoine/mambaforge/envs/pyarrow/lib/libarrow.so.1300.0.0  lib/libgtestd.a  -lpthread && cd /home/antoine/arrow/nanoarrow/build-debug && /home/antoine/mambaforge/envs/pyarrow/bin/cmake -D TEST_TARGET=utils_test -D TEST_EXECUTABLE=/home/antoine/arrow/nanoarrow/build-debug/utils_test -D TEST_EXECUTOR= -D TEST_WORKING_DIR=/home/antoine/arrow/nanoarrow/build-debug -D TEST_EXTRA_ARGS= -D TEST_PROPERTIES= -D TEST_PREFIX= -D TEST_SUFFIX= -D TEST_FILTER= -D NO_PRETTY_TYPES=FALSE -D NO_PRETTY_VALUES=FALSE -D TEST_LIST=utils_test_TESTS -D CTEST_FILE=/home/antoine/arrow/nanoarrow/build-debug/utils_test[1]_tests.cmake -D TEST_DISCOVERY_TIMEOUT=5 -D TEST_XML_OUTPUT_DIR= -P /home/antoine/mambaforge/envs/pyarrow/share/cmake-3.26/Modules/GoogleTestAddTests.cmake
/home/antoine/mambaforge/envs/pyarrow/bin/../lib/gcc/x86_64-conda-linux-gnu/13.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libcrypto.so.3, needed by /home/antoine/mambaforge/envs/pyarrow/lib/libarrow.so.1300.0.0, not found (try using -rpath or -rpath-link)
[...]
/home/antoine/mambaforge/envs/pyarrow/bin/../lib/gcc/x86_64-conda-linux-gnu/13.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/antoine/mambaforge/envs/pyarrow/lib/libarrow.so.1300.0.0: undefined reference to `BrotliEncoderCompressStream'
[etc.]

nanoarrow is configured using:

cmake .. -DCMAKE_BUILD_RPATH=${CONDA_PREFIX}/lib -DCMAKE_BUILD_TYPE=debug -DNANOARROW_BUILD_TESTS=on -GNinja

Note that all libraries do resolve from the command line:

$ lddtree /home/antoine/mambaforge/envs/pyarrow/lib/libarrow.so.1300.0.0
libarrow.so.1300.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libarrow.so.1300.0.0 (interpreter => none)
    libcrypto.so.3 => /home/antoine/mambaforge/envs/pyarrow/lib/libcrypto.so.3
    libssl.so.3 => /home/antoine/mambaforge/envs/pyarrow/lib/libssl.so.3
    libbrotlienc.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libbrotlienc.so.1
    libbrotlidec.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libbrotlidec.so.1
    libbrotlicommon.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libbrotlicommon.so.1
    liborc.so => /home/antoine/mambaforge/envs/pyarrow/lib/liborc.so
    libprotobuf.so.32 => /home/antoine/mambaforge/envs/pyarrow/lib/libprotobuf.so.32
    libutf8proc.so.2 => /home/antoine/mambaforge/envs/pyarrow/lib/libutf8proc.so.2
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
    libgoogle_cloud_cpp_storage.so.2 => /home/antoine/mambaforge/envs/pyarrow/lib/libgoogle_cloud_cpp_storage.so.2
    libbz2.so.1.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libbz2.so.1.0
    liblz4.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/liblz4.so.1
    libsnappy.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libsnappy.so.1
    libz.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libz.so.1
    libzstd.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libzstd.so.1
    libaws-cpp-sdk-config.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-config.so
    libaws-cpp-sdk-transfer.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-transfer.so
    libaws-cpp-sdk-identity-management.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-identity-management.so
    libaws-cpp-sdk-cognito-identity.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-cognito-identity.so
    libaws-cpp-sdk-sts.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-sts.so
    libaws-cpp-sdk-s3.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-s3.so
    libaws-cpp-sdk-core.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-cpp-sdk-core.so
    libre2.so.10 => /home/antoine/mambaforge/envs/pyarrow/lib/libre2.so.10
    libabsl_str_format_internal.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_str_format_internal.so.2206.0.0
    libgoogle_cloud_cpp_rest_internal.so.2 => /home/antoine/mambaforge/envs/pyarrow/lib/libgoogle_cloud_cpp_rest_internal.so.2
    libgoogle_cloud_cpp_common.so.2 => /home/antoine/mambaforge/envs/pyarrow/lib/libgoogle_cloud_cpp_common.so.2
    libabsl_time.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_time.so.2206.0.0
    libabsl_strings.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_strings.so.2206.0.0
    libabsl_strings_internal.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_strings_internal.so.2206.0.0
    libabsl_int128.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_int128.so.2206.0.0
    libabsl_civil_time.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_civil_time.so.2206.0.0
    libabsl_time_zone.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_time_zone.so.2206.0.0
    libabsl_bad_variant_access.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_bad_variant_access.so.2206.0.0
    libabsl_base.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_base.so.2206.0.0
    libabsl_spinlock_wait.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_spinlock_wait.so.2206.0.0
    libabsl_bad_optional_access.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_bad_optional_access.so.2206.0.0
    libabsl_throw_delegate.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_throw_delegate.so.2206.0.0
    libabsl_raw_logging_internal.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_raw_logging_internal.so.2206.0.0
    libabsl_log_severity.so.2206.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libabsl_log_severity.so.2206.0.0
    libcrc32c.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libcrc32c.so.1
    libcurl.so.4 => /home/antoine/mambaforge/envs/pyarrow/lib/libcurl.so.4
        libnghttp2.so.14 => /home/antoine/mambaforge/envs/pyarrow/lib/./libnghttp2.so.14
        libssh2.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/./libssh2.so.1
        libgssapi_krb5.so.2 => /home/antoine/mambaforge/envs/pyarrow/lib/./libgssapi_krb5.so.2
            libkrb5.so.3 => /home/antoine/mambaforge/envs/pyarrow/lib/././libkrb5.so.3
            libk5crypto.so.3 => /home/antoine/mambaforge/envs/pyarrow/lib/././libk5crypto.so.3
            libcom_err.so.3 => /home/antoine/mambaforge/envs/pyarrow/lib/././libcom_err.so.3
            libkrb5support.so.0 => /home/antoine/mambaforge/envs/pyarrow/lib/././libkrb5support.so.0
            libkeyutils.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/././libkeyutils.so.1
            libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
    libaws-crt-cpp.so => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-crt-cpp.so
    libaws-c-mqtt.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-mqtt.so.1.0.0
    libaws-c-event-stream.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-event-stream.so.1.0.0
    libaws-c-s3.so.0unstable => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-s3.so.0unstable
    libaws-c-auth.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-auth.so.1.0.0
    libaws-c-http.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-http.so.1.0.0
    libaws-c-io.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-io.so.1.0.0
    libs2n.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libs2n.so.1
    libaws-c-compression.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-compression.so.1.0.0
    libaws-c-cal.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-cal.so.1.0.0
    libaws-c-sdkutils.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-sdkutils.so.1.0.0
    libaws-checksums.so.1.0.0 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-checksums.so.1.0.0
    libaws-c-common.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libaws-c-common.so.1
    libstdc++.so.6 => /home/antoine/mambaforge/envs/pyarrow/lib/libstdc++.so.6
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libgcc_s.so.1 => /home/antoine/mambaforge/envs/pyarrow/lib/libgcc_s.so.1
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

[pitrou]

@kou Do you have any idea how to solve this?

[pitrou]

Note that setting LD_LIBRARY_PATH works around the issue.

[paleolimbot]

Thank you for reporting!

I can reproduce this on Ubuntu 22.04 with:

conda create --name nanoarrow-verify-rc
conda activate nanoarrow-verify-rc
conda config --set channel_priority strict

mamba install -c conda-forge compilers cmake make git xsimd
curl https://dlcdn.apache.org/arrow/arrow-12.0.1/apache-arrow-12.0.1.tar.gz | \
  tar -zxf -
mkdir arrow-build && cd arrow-build
cmake ../apache-arrow-12.0.1/cpp -DARROW_FILESYSTEM=ON -DARROW_WITH_ZLIB=ON -DARROW_DEPENDENCY_SOURCE=CONDA
cmake --build . --parallel=$(nproc)
cmake --install . --prefix=${CONDA_PREFIX}
cd ..

git clone https://github.com/apache/arrow-nanoarrow.git arrow-nanoarrow
cd arrow-nanoarrow
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -DNANOARROW_BUILD_TESTS=ON -DCMAKE_BUILD_RPATH=${CONDA_PREFIX}/lib
cmake --build .

[kou]

(I'm creating an environment to reproduce this...)

[kou]

I couldn't reproduce this on Ubuntu 22.04 based on https://app.vagrantup.com/bento/boxes/ubuntu-22.04 ...
I used https://repo.anaconda.com/miniconda/Miniconda3-py311_23.5.2-0-Linux-x86_64.sh instead of mamba.
I used command lines provided by #254 (comment) but I replaced mamba with conda.

[paleolimbot]

I spent some more time on this and found that if you restart the terminal, activate the requisite conda environment, and do a fresh configure + build of nanoarrow, the problem goes away (however, if you don't do a fresh configure + build of nanoarrow, the linking error persists). It appears that you can't build + install Arrow C++ from the same terminal session (or perhaps conda activation) where you configure and build nanoarrow that links to Arrow C++.

As a workaround in case this doesn't fix the problem, #267 lets you pass -DNANOARROW_ARROW_STATIC=ON and statically link Arrow into the tests. We did this anyway on Windows...the PR just generalized the option for times like this.

[pitrou]

It appears that you can't build + install Arrow C++ from the same terminal session (or perhaps conda activation) where you configure and build nanoarrow that links to Arrow C++.

I build Arrow C++ and nanoarrow from different terminal instances, so I doubt this is the issue?

[vyasr]

I haven't been able to reproduce this. I've tried it out with both current and a couple of older versions of arrow, but using the latest nanoarrow. Would it make sense to close this for now but reopen if and when the issue reoccurs? I'm happy to take a look at it when that happens since I'm doing a bit of work right now to improve nanoarrow's CMake anyway.