mxnet 1.9.1 cuda support for Jetson AGX Orin (Jetpack 5.0 with cuda 11.4)

[anilrgukt]

Description

mxnet (1.9.1) is raising a cuda error with cuda 11.4 on Jetson AGX Orin series with Jetpack 5.0

Error Message

Check failed: err == cudaSuccess (209 vs. 0) : mxnet_generic_kernel ErrStr:no kernel image is available for execution on the device

To Reproduce

I ran the installation steps recommended for the jetson and built things manually from source with cuda path and other variables.

Everything went fine with the make and importing mxnet and version check. When I run the gpu check the error pops up.

What have you tried to solve it?

1. Verified cuda installation, no errors raise during make
2. Other libraries like pytorch GPU are working fine

Environment

Ubuntu 20.04, Nvidia jetson AGX Orin developer kit

Environment Information

----------Python Info----------
Version      : 3.8.10
Compiler     : GCC 9.4.0
Build        : ('default', 'Nov 14 2022 12:59:47')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 22.3.1
Directory    : /home/flashsys002/py38/lib/python3.8/site-packages/pip
----------MXNet Info-----------
Version      : 1.9.1
Directory    : /home/flashsys002/mxnet/python/mxnet
Commit hash file "/home/flashsys002/mxnet/python/mxnet/COMMIT_HASH" not found. Not installed from pre-built package or built from source.
Library      : ['/home/flashsys002/mxnet/lib/libmxnet.so', '/home/flashsys002/mxnet/python/mxnet/../../lib/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✖ NCCL
✖ CUDA_RTC
✖ TENSORRT
✖ CPU_SSE
✖ CPU_SSE2
✖ CPU_SSE3
✖ CPU_SSE4_1
✖ CPU_SSE4_2
✖ CPU_SSE4A
✖ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✖ F16C
✖ JEMALLOC
✔ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✖ LAPACK
✖ MKLDNN
✖ OPENCV
✖ CAFFE
✖ PROFILER
✖ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform     : Linux-5.10.65-tegra-aarch64-with-glibc2.29
system       : Linux
node         : flashsys002-desktop
release      : 5.10.65-tegra
version      : #1 SMP PREEMPT Mon May 16 20:58:07 PDT 2022
----------Hardware Info----------
machine      : aarch64
processor    : aarch64
Architecture:                    aarch64  
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       3
Vendor ID:                       ARM
Model:                           1
Model name:                      ARMv8 Processor rev 1 (v8l)
Stepping:                        r0p1
CPU max MHz:                     2201.6001
CPU min MHz:                     115.2000 
BogoMIPS:                        62.50
L1d cache:                       768 KiB  
L1i cache:                       768 KiB  
L2 cache:                        3 MiB
L3 cache:                        6 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/mxnet, DNS: 0.0029 sec, LOAD: 0.4289 sec.
Error open Gluon Tutorial(en): http://gluon.mxnet.io, HTTP Error 404: Not Found, DNS finished in 0.07585453987121582 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)>, DNS finished in 0.035755157470703125 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0161 sec, LOAD: 0.4418 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0113 sec, LOAD: 0.2129 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.021700382232666016 sec.
----------Environment----------
MXNET_HOME="/home/flashsys002/mxnet/"
KMP_DUPLICATE_LIB_OK="True"
KMP_INIT_AT_FORK="FALSE"

[github-actions[bot]]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[anilrgukt]

It works now, I had to modify the arch from 72 to 87 for Orin modules -gencode arch=compute_72,code=sm_72 for Xavier NX and AGX Xavier in the config make file