Compile GPU version on OSC

[Wang-Zhongwei]

Hi WarpX Developers,

I have successfully compiled the CPU version of WarpX on the Ohio Supercomputer Center (OSC). However, I encountered a relocation truncation error during the linking phase of the GPU build. I believe this error is due to the large code size, as explained here. The total size of build_gpu, which contains only 1D code, is 5.6GB. Is that too large? A detailed output of the error can be found in this log file: compile_gpu_29925223.log.

Here is my building script:

module reset
ml gnu/9.1.0
ml cuda/11.8.0
ml cmake/3.25.2
ml openmpi/4.0.3-hpcx
ml mkl/2019.0.5
ml hdf5/1.12.0

export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=${CXX}

export CUDAFLAGS="--host-linker-script=use-lcs" # https://github.com/ECP-WarpX/WarpX/pull/3673 
export AMReX_CUDA_ARCH=7.0 # 7.0: V100, 8.0: A100, 9.0: H100 https://github.com/ECP-WarpX/WarpX/issues/3214
cmake -S . -B build_gpu -DWarpX_DIMS="1" -DWarpX_COMPUTE=CUDA 
cmake --build build_gpu -j 48

I attempted to resolve the relocation truncation error by following the suggestions in #3673 and #3214, but the error persists.

Thank you for your time!

[ax3l]

have successfully compiled the CPU version of WarpX on the Ohio Supercomputer Center (OSC).

Awesome, congratulations!
Once we found a working setup, let's add the instructions you found to https://warpx.readthedocs.io/en/latest/install/hpc.html via a pull request to WarpX? 🎉

However, I encountered a relocation truncation error during the linking phase of the GPU build. I believe this error is due to the large code size, as explained here. The total size of build_gpu, which contains only 1D code, is 5.6GB. Is that too large?

Ah, this is an error we sometimes see with some compilers. Yes, the approach to set

export CUDAFLAGS="--host-linker-script=use-lcs"

could solve it - did you run rm -rf build_gpu before running CMake again? Otherwise it will not pick up that flag.

[Wang-Zhongwei]

Sure, I will add compilation and running guide for OSC and make a pull request later today!

[ax3l]

Alternatively, you can try using a newer GNU GCC compiler, e.g., 10 or 11 also work with CUDA 11.8: https://gist.github.com/ax3l/9489132

[ax3l]

Do not load ml mkl/2019.0.5, we should not need MKL and it could interfere in some way with the other GCC-based modules.

[Wang-Zhongwei]

Thank you! I think the problem was about this module. After I removed it, I successfully compiled the build_gpu with storage less than 1GB

[ax3l]

One of your lines contains a typo: our environment variables need to be all-CAPS, so it should read:

export AMREX_CUDA_ARCH=7.0

(note the E in AMREX.)
https://amrex-codes.github.io/amrex/docs_html/GPU.html?highlight=amrex_cuda_arch#enabling-cuda-support