Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM)-based stacks

[rickgrubin-noaa]

Description

Build and test UFS WM against spack-stack@release/1.8.0 built with Intel oneAPI v2024.2.1 (LLVM) compilers.

Solution

stack environments

hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs

• ifx

  • ue-llvm-ifx-2024.2.1
  • gsi-llvm-ifx-2024.2.1

• ifort

  • ue-llvm-ifort-2024.2.1
  • gsi-llvm-ifort-2024.2.1

Will also require addon environments for each compiler for a known-to-work combination of MAPL / ESMF.

Once the MAPL / ESMF addon envs are ready, steps to load required modules, e.g. for the ifx compiler:

module purge
module use /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/modulefiles/Core
module load stack-oneapi/2024.2.1
module load stack-intel-oneapi-mpi/2021.13
module load ufs-weather-model-env/1.0.0

Build and test UFS WM with appropriate changes to modules loaded to reflect the above stacks.

Additional Context

This issue is to track the UFS WM issue Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM) compilers #2513

[rickgrubin-tomorrow]

[ Ref: internal EPIC ticket; note also that results / path below are specific to hera ]

UFS WM Regression Testing results

ifort

• [email protected] (successful compile (non-DEBUG) and successful model run)
  • [email protected] and [email protected] and [email protected]
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2455064

ifx

• [email protected] (successful compile (non-DEBUG) and failed model run)
  • [email protected] and [email protected] and [email protected]
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2949966
• [email protected] (failed compile (DEBUG))
  • [email protected] and [email protected] and [email protected]
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2377353
• [email protected] (successful compile (non-DEBUG) and failed model run)
  • [email protected] and [email protected] and [email protected]
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_1323858

For this case: [email protected] (failed compile (DEBUG))

[...]
[100%] Linking Fortran executable ufs_model
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/cmake-3.27.9-ihjoyd3/bin/cmake -E cmake_link_script CMakeFiles/ufs_model.dir/link.txt --verbose=1
/apps/oneapi/mpi/2021.13/bin/mpifc -fiopenmp  -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -align array64byte -qno-opt-dynamic-align -diag-disable 5462 -diag-disable 7712 -diag-disable 8291 -diag-disable 5194 -g -O0 -check -check noarg_temp_created -check nopointer -warn -warn noerrors -fp-stack-check -fstack-protector-all -fpe0 -debug minimal -ftrapuv -init=snan,arrays CMakeFiles/ufs_model.dir/driver/UFS.F90.o -o ufs_model 
[...]
-L/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib -lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -m64 -mcmodel=small -pthread -threads
[...]

ld: cannot find -lpioc
ld: attempted static link of dynamic object `/apps/oneapi/compiler/2024.2/lib/libiomp5.so'

and

% file /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

% ldd /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallelio-2.6.2-prs7utx/lib/libpioc.so
	linux-vdso.so.1 (0x00007ffea02e0000)
	libnetcdf.so.19 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/netcdf-c-4.9.2-ulez3hv/lib/libnetcdf.so.19 (0x00007f8c050b6000)
	libpnetcdf.so.4 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/parallel-netcdf-1.12.3-3olk4vh/lib/libpnetcdf.so.4 (0x00007f8c0445e000)
	libmpifort.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpifort.so.12 (0x00007f8c040b2000)
	libmpi.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpi.so.12 (0x00007f8bfa277000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f8bfa073000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f8bf9e6b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8bf9c4b000)
	libsvml.so => /apps/oneapi/compiler/2024.2/lib/libsvml.so (0x00007f8bf8607000)
	libirng.so => /apps/oneapi/compiler/2024.2/lib/libirng.so (0x00007f8c05869000)
	libimf.so => /apps/oneapi/compiler/2024.2/lib/libimf.so (0x00007f8bf81fb000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f8bf7e79000)
	libintlc.so.5 => /apps/oneapi/compiler/2024.2/lib/libintlc.so.5 (0x00007f8c05806000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8bf7aa3000)
	libhdf5_hl.so.310 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/hdf5-1.14.3-5szfql6/lib/libhdf5_hl.so.310 (0x00007f8bf7880000)
	libhdf5.so.310 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/hdf5-1.14.3-5szfql6/lib/libhdf5.so.310 (0x00007f8bf71a9000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f8bf6f98000)
	libzstd.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/zstd-1.5.2-clennzi/lib/libzstd.so.1 (0x00007f8bf6cae000)
	libblosc.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/c-blosc-1.21.5-rwudonl/lib64/libblosc.so.1 (0x00007f8bf6a84000)
	libxml2.so.2 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/libxml2-2.10.3-tnoi22x/lib/libxml2.so.2 (0x00007f8bf66eb000)
	libcurl.so.4 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/curl-8.7.1-ilyildl/lib/libcurl.so.4 (0x00007f8bf6427000)
	libifport.so.5 => /apps/oneapi/compiler/2024.2/lib/libifport.so.5 (0x00007f8c057d8000)
	libifcoremt.so.5 => /apps/oneapi/compiler/2024.2/lib/libifcoremt.so.5 (0x00007f8bf62bb000)
	libirc.so => /apps/oneapi/compiler/2024.2/lib/libirc.so (0x00007f8bf625a000)
	libstdc++.so.6 => /apps/gnu/gcc-9.2.0/lib64/libstdc++.so.6 (0x00007f8bf5e7b000)
	libgcc_s.so.1 => /apps/gnu/gcc-9.2.0/lib64/libgcc_s.so.1 (0x00007f8bf5c63000)
	libmpicxx.so.12 => /apps/oneapi/mpi/2021.13/lib/libmpicxx.so.12 (0x00007f8bf5a43000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8c0574a000)
	liblz4.so.1 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/lz4-1.9.4-mt5apsb/lib/liblz4.so.1 (0x00007f8bf5816000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8bf55ef000)
	libnghttp2.so.14 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/nghttp2-1.57.0-w273rt4/lib/libnghttp2.so.14 (0x00007f8bf53bf000)
	libssl.so.3 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/openssl-3.3.0-2cqpman/lib64/libssl.so.3 (0x00007f8bf50c4000)
	libcrypto.so.3 => /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/oneapi/2024.2.1/openssl-3.3.0-2cqpman/lib64/libcrypto.so.3 (0x00007f8bf4aa0000)



% file /apps/oneapi/compiler/2024.2/lib/libiomp5.so
/apps/oneapi/compiler/2024.2/lib/libiomp5.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

%  ldd /apps/oneapi/compiler/2024.2/lib/libiomp5.so
	linux-vdso.so.1 (0x00007ffe8988d000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f888c09f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f888be9b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f888bc7b000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f888b8a5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f888c68f000)

For the simple test run (control_c48), the DEBUG settings are:

COMPILE | atm_dyn32 | intelllvm | -DAPP=ATM -DDEBUG=ON -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn,FV3_GFS_v17_p8_ugwpv1 -D32BIT=ON | | fv3 |
RUN | control_c48                                       |                                      | baseline |