Unit tests fail on Frontier after recent module updates

[philipwjones]

Recent updates of modules on Frontier are preventing successful builds, so the E3SM config_machine must be updated.

As a workaround, the default compiler configurations appear to work for the crayclang (CPU) and crayomegagpu (GPU) options:
PrgEnv-amd/8.5.0
PrgEnv-cray-amd/8.5.0
PrgEnv-cray/8.5.0
PrgEnv-gnu-amd/8.5.0
PrgEnv-gnu/8.5.0
amd/5.7.1
craype/2.7.31
cce/17.0.0
cpe/23.12
gcc/12.2.0
rocm/5.7.1

cray-netcdf-hdf5parallel/4.9.0.9
cray-hdf5-parallel/1.12.2.9
cray-parallel-netcdf/1.12.3.9

I don't know if these options will work for the E3SM builds, so the eventual solution may be different.

[philipwjones]

Workaround posted here as promised on today's omega call @sbrus89 @xylar

[xylar]

Thanks @philipwjones!

[sbrus89]

Thanks @philipwjones!

[philipwjones]

I recently inadvertantly built omega with the old modules to test a PR and they appear to all be working again - not sure if OLCF reinstalled old mods for back-compatibility? We should still update to more recent versions, but workaround may no longer be necessary.

[grnydawn]

@philipwjones, thanks for the info. I was also able to successfully build and run Omega without updating the modules. In any case, I will keep working on updating the e3sm machine config and then update the Omega config.

[sbrus89]

@philipwjones and @grnydawn are either of you still able to build/run Omega without updating modules on Frontier? I tried this today and was able to build successfully, but am seeing a bunch of /opt/cray/pe/gcc-libs/libstdc++.so.6: version 'GLIBCXX_3.4.32' not found-related errors when running the ctests.

[philipwjones]

@sbrus89 Yup, still works for me. I usually build with crayclang for cpu and crayomegagpu for gpu. Both successfully built and ran the develop branch with both old and new modules.

[grnydawn]

@sbrus89 , I could not reproduce it because I could not get allocation on Frontier today. If you can share the location of the Omega build directory with me, I will look into it.

[mwarusz]

@sbrus89
I got the same error on gpu when I tried testing #105 on Frontier yesterday. I was using crayclanggpu. Cpu build with gnu worked fine.

[grnydawn]

@sbrus89, @mwarusz, try to use "crayomegagpu" instead of "crayclanggpu." Until Omega is fully compatible with E3SM configurations, we want to use Omega-specific configurations including "crayomegagpu," "gnuomegagpu," and "amdomegagpu."

[sbrus89]

Thanks @philipwjones and @grnydawn!