onnxruntime-gpu cannot use cuda libs that installed via pip?

[braindevices]

how could I get it to work?

session = ort.InferenceSession(model_path, providers=["CUDAExecutionProvider"])
session.get_providers()

onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12

other package seems to be able to use cublas from the pip installed version:

+ exec pip list
nvidia-cublas-cu12               12.1.3.1
nvidia-cuda-cupti-cu12           12.1.105
nvidia-cuda-nvrtc-cu12           12.1.105
nvidia-cuda-runtime-cu12         12.1.105
nvidia-cudnn-cu12                9.1.0.70
nvidia-cufft-cu12                11.0.2.54
nvidia-curand-cu12               10.3.2.106
nvidia-cusolver-cu12             11.4.5.107
nvidia-cusparse-cu12             12.1.0.106
nvidia-nccl-cu12                 2.20.5
nvidia-nvjitlink-cu12            12.5.40
nvidia-nvtx-cu12                 12.1.105

[tianleiwu]

nvidia-cublas-cu12 contains the libcublasLt.so.12 and libcublas.so.12.

Please run the following shell script to update LD_LIBRARY_PATH:

site_packages_dir="$(python3 -c "import sysconfig; print(sysconfig.get_path('purelib'))")"
dirs=$(find "$site_packages_dir/nvidia" -type f \( -name "*.so" -o -name "*.so.*" \) -exec dirname {} \; | sort -u)
for dir in $dirs; do
    if [[ ":$LD_LIBRARY_PATH:" != *":$dir:"* ]]; then
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$dir"
    fi
done
echo "$LD_LIBRARY_PATH"

After that you can try your python script again.

You can also do it in python like

import os
import sys
from glob import glob

def find_dirs_with_shared_objects(base_dir):
    # Patterns to match .so, .so.1, .so.11, .so.12 files
    so_patterns = ["*.so", "*.so.1", "*.so.11", "*.so.12"]

    dirs_with_so = set()

    # Search for files matching the patterns
    for pattern in so_patterns:
        files = glob(os.path.join(base_dir, "**", pattern), recursive=True)
        for file_path in files:
            dirs_with_so.add(os.path.dirname(file_path))

    return list(dirs_with_so)

def add_directories_to_ld_library_path(dirs):
    # For non-Windows systems (Linux/macOS), update LD_LIBRARY_PATH
    if sys.platform != "win32":
        ld_library_path = os.environ.get("LD_LIBRARY_PATH", "")
        new_ld_library_path = ld_library_path + ":" + ":".join(dirs) if ld_library_path else ":".join(dirs)
        os.environ["LD_LIBRARY_PATH"] = new_ld_library_path
        print(f"Updated LD_LIBRARY_PATH: {os.environ['LD_LIBRARY_PATH']}")
    else:
        # For Windows, use os.add_dll_directory
        for dir in dirs:
            os.add_dll_directory(dir)
        print(f"Added directories using os.add_dll_directory")

def main():
    # Get the site-packages/nvidia directory in the current Python environment
    site_packages_dir = os.path.join(sys.prefix, "lib", "python{0}.{1}".format(*sys.version_info[:2]), "site-packages", "nvidia")

    if not os.path.exists(site_packages_dir):
        print(f"Directory {site_packages_dir} does not exist.")
        return

    # Find directories containing .so, .so.11, or .so.12 files
    dirs_with_so_files = find_dirs_with_shared_objects(site_packages_dir)

    if not dirs_with_so_files:
        print("No directories with .so or .so.11 or .so.12 files found.")
        return

    # Add those directories to LD_LIBRARY_PATH or os.add_dll_directory
    add_directories_to_ld_library_path(dirs_with_so_files)

if __name__ == "__main__":
    main()