[Experimental] interrogator_rpc + cu12

[thiagojramos]

For anyone wanting to use interrogator_rpc with (PyTorch) CUDA 12.x, first install the latest torch with CUDA using pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121, and then install onnxruntime-gpu via ORT Azure DevOps Feed. There are two methods for this:

Method 01

Download the wheels available on the page and install via pip install "path_to_whl"

Method 02

As shown in the image below, go to your "pip.ini", which is usually located in "%APPDATA%\pip", and add (or create if it doesn’t exist) the following line under "extra-index-url =":

https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

Here is an example of my pip.ini:

[global]
index-url = https://pypi.org/simple
extra-index-url =
                  https://pypi.ngc.nvidia.com
                  https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
trusted-host =
               pypi.ngc.nvidia.com
               aiinfra.pkgs.visualstudio.com

In my case, the file was generated by something from NVIDIA NGC, so I just needed to add the new line. Then, go to the terminal and type pip install onnxruntime-gpu or
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
If you later want to install the standard onnxruntime-gpu (for CUDA 11.8), just remove this line from pip.ini.

---

Now, just follow the installation for onnx, grpcio, grpcio-tools, pandas, and transformers.

Despite the yellow warning message, I didn’t notice any performance drops, freezes, or any negative impact. As you can see in the image, the tags were successfully generated (it took about 1 or 2 seconds).

Additional Information

Remember to install and have CUDA 12.x and CuDNN v8.9.7 in your system path (unfortunately, it doesn't seem to work with CuDNN v9 or later).

As mentioned in the CuDNN v8.9.7 documentation, you also need to have the folder containing zlibwapi.dll in your system path. You can obtain Zlib DLL from this link (x64).