Hoomd GPU Error

[entrylevelcs]

I am having an problem running simulations with Hoomd on a GPU cluster. It seems to build from source just fine but when I run simulations I get a runtime error telling me that there are no supported GPUs present on the system. Attached will be some system and environment information that I am using.
extra_information.txt

[joaander]

Please provide the output of nvidia-smi.

Also, build the num-devices-errors branch (#1803) and hoomd should report a more detailed error message than "Unknown error."

[entrylevelcs]

Here is the output:

I will build from that branch and paste the error message here after I run it.

[entrylevelcs]

Rebuilding it on the num-devices-errors branch I still end up with the same error as before

[joaander]

Please copy and paste text as text. It is very difficult to read blurry screenshots and I am unable to copy and paste relevant information to highlight.

paste terminal output and code snippets like this. They are easier to see than in an attached image or text file.

See the GitHub documentation for details.

[joaander]

Can you verify that you are using the correct branch?

python3 -c "import hoomd; print(hoomd.version.git_sha1)"

should print:

845713b9b8a4f1c4f9cb244def9aa080b7ccef2e

I would have expected an error something like:

"Insufficient driver"

or if the error is truly unknown, CUDA states that it reports

"unrecognized error code"

[joaander]

Seeing as you have CUDA 10.0 drivers and are building HOOMD with CUDA 12, I would assume that you are getting the [cudaErrorInsufficientDriver error] (https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038)

This indicates that the installed NVIDIA CUDA driver is older than the CUDA runtime library. This is not a supported configuration. Users should install an updated NVIDIA display driver to allow the application to run.

Have you tried upgrading the driver or building HOOMD with a compatible version of CUDA? You should have. I suggested that this was likely the problem in my original e-mail reply to your direct message.

[entrylevelcs]

Running python3 -c "import hoomd; print(hoomd.version.git_sha1)" I get 845713b9b8a4f1c4f9cb244def9aa080b7ccef2e as the output so it appears that I am building hoomd from that branch but I am still getting the same error as before. We are not able to update the cuda version of our cluster so I have been trying to build hoomd with different versions of CUDA but have not yet found any success. I will continue to try other versions CUDA to see if the problem persists.

[joaander]

I currently test and validate HOOMD-blue with CUDA 11 and 12 only, so updating the drivers may be your only option.