Non-zero status code returned while running MatMul node.

[vahmelk99]

Hi everyone. I'm trying to run a pretrained tracking model on CUDA which uses onnx. When I'm calling onnxruntime.InferenceSession with CUDAExecutionProvider like this

self.ort_session_z = onnxruntime.InferenceSession(backbone_path, providers=['CUDAExecutionProvider'])
self.ort_session_x = onnxruntime.InferenceSession(model_path, providers=['CUDAExecutionProvider'])

it is showing an error and not working on CUDA.
The error is the following:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : 
Non-zero status code returned while running MatMul node. Name:'MatMul_188' Status Message: 
CUBLAS failure 7: CUBLAS_STATUS_INVALID_VALUE ; GPU=0 ; hostname=******; 
expr=cublasGemmHelper( GetCublasHandle(ctx), transB, transA, static_cast<int>(helper.N()), 
static_cast<int>(helper.M()), static_cast<int>(helper.K()), &alpha, 
reinterpret_cast<const CudaT*>(right_X->Data<T>()), ldb, reinterpret_cast<const CudaT*>(left_X->Data<T>()), 
lda, &zero, reinterpret_cast<CudaT*>(Y->MutableData<T>()), ldc, device_prop); 

When passing providers CPUExecutionProvider or TensorrtExecutionProvider it is running on CPU.

My Setup:

Ubuntu 18.04.6
NVIDIA GeForce RTX 3060
torch 1.13.1
onnx 1.14.1
onnxruntime-gpu 1.14.0
nvcc -V: Build cuda_11.6.r11.6/compiler.30794723_0
nvidia-smi 545.29.02 Driver Version

[vahmelk99]

After writing my question I found the solution)
In case someone has a similar issue here's the solution and also where did I find that solution.

 pip3 install nvidia-pyindex
 pip3 install nvidia-cublas