[Build]

[Bahar-BM]

Describe the issue

Hello,

I’ve successfully built ONNX Runtime 1.19.2 for CUDA 11.8 on an aarch64 architecture. However, when I try to run my application, I encounter the following runtime error:

CUDA failure 222: the provided PTX was compiled with an unsupported toolchain

Upon investigating this issue further, I discovered that the .target sm_52 specified in build/Linux/Release/CMakeFiles/3.26.0/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx is incorrect. The compute capability of my device is actually 8.7.

My question is: does the build system use a default compute capability? If so, how can I change it to match my device’s compute capability?

Urgency

No response

Target platform

tegra-ubuntu aarch64

Build script

I'm using build.sh with the following args
./build.sh --config Release --build --build_shared_lib --use_cuda --cuda_home /path/to/cuda-11.8 --cudnn_home /usr/lib/aarch64-linux-gnu

Error / output

'std::runtime_error'
what(): /mnt/ssd_drive/repos/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] /mnt/ssd_drive/repos/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] CUDA failure 222: the provided PTX was compiled with an unsupported toolchain. ; GPU=0 ; hostname=tegra-ubuntu ; file=/mnt/ssd_drive/repos/onnxruntime/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=283 ; expr=cudaDeviceSynchronize();

Aborted

Visual Studio Version

No response

GCC / Compiler Version

10

[snnn]

Please add "--cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=87" to your build command.

See https://github.com/microsoft/onnxruntime/blob/main/tools/ci_build/github/linux/build_cuda_ci.sh for an example.

[tianleiwu]

If "--cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=87" does not work, you will need upgrade your cuda driver. User has reported similar issue (#22212) due to cuda driver not up to date.

[Bahar-BM]

Hello,

The --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=87 option did not work. I even tried to force the CMake build process to pick 87 during the configuration phase by hardcoding set(CMAKE_CUDA_ARCHITECTURES 87). However, it did not work.

As you mentioned, there is a high chance the issue is related to the CUDA drivers.

Thanks!