Updates to build and run on Orion Rocky 9

[RussTreadon-NOAA]

Description
This PR updates NOAA-EMC/GSI to build and run on Orion Rocky 9.

Resolves #754

Type of change

• Maintenance

How Has This Been Tested?
Install on Orion and run ctests with results (all tests Passed) posted in issue #754.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• New and existing tests pass with my changes

[RussTreadon-NOAA]

Open PR in draft mode given increased wall times observed for gsi.x and enkf.x when run on Orion Rocky 9.

[RussTreadon-NOAA]

Slowness of gsi.x and enkf.x on Orion Rocky 9 remains unexplained but will change this PR to Ready for review to invite feedback on proposed changes.

[DavidHuber-NOAA]

Looks good to me. Hopefully between spack-stack developers and Orion admins we can sort out the slowdown on Orion.

[RussTreadon-NOAA]

Agreed, @DavidHuber-NOAA ! The gsi.x and enkf.x slowdown on Orion Rocky 9 does not make sense to me.

[aerorahul]

@RussTreadon-NOAA
Can this PR be merged with the knowledge that GSI is running in degraded status on Orion?
This will allow the global-workflow to proceed for Orion+Rocky8.
When the source of the degradation is identified on Orion, we can update the submodule pointer for Orion+Rocky8

Tagging @CatherineThomas-NOAA for awareness and discussions on the GFSv17 project tag-up.

[RussTreadon-NOAA]

@aerorahul , let me check with the GSI Review team

@ShunLiu-NOAA , @CoryMartin-NOAA , and @hu5970 : Are we OK merging this PR into GSI develop even though ctests show gsi.x and enkf.x run approximately 2x slower on Orion Rocky 9 and Orion Centos 7? @aerorahul explains above why this question is being asked.

I'm reluctant to merge since do so may lessen the urgency of addressing the 2x slowdown. That said, we don't want NOAA-EMC/GSI to become the roadblock for completion of the g-w transition to Orion Rocky 9 (see issue #2694)

[CoryMartin-NOAA]

@RussTreadon-NOAA I share your concerns but I don't think we have a choice. It's either "runs slow" or "not at all", so I think we have to go with the former.

[aerorahul]

@aerorahul , let me check with the GSI Review team

@ShunLiu-NOAA , @CoryMartin-NOAA , and @hu5970 : Are we OK merging this PR into GSI develop even though ctests show gsi.x and enkf.x run approximately 2x slower on Orion Rocky 9 and Orion Centos 7? @aerorahul explains above why this question is being asked.

I'm reluctant to merge since do so may lessen the urgency of addressing the 2x slowdown. That said, we don't want NOAA-EMC/GSI to become the roadblock for completion of the g-w transition to Orion Rocky 9 (see issue #2694)

Just open another issue to report the performance degradation on Orion after the upgrade and follow the development there.

[RussTreadon-NOAA]

Actions already taken reporting gsi.x and enkf.x slowdown on Orion Rocky-9:

• open ticket RDHPCS ticket #2024062754000098 with Orion Helpdesk
• open spack-stack issue #1166

Someone will need to follow up on the ticket and issue. doing so will likely require trying various things until the problem is resolved.