Add weekly build/test scripts

[AlexanderRichert-NOAA]

Summary

This PR will add scripting for weekly unit testing and build caching of the unified env through Jenkins, cron jobs, etc.

Also a few updates for Acorn, including a change to spack stack create env --compiler that allows compiler version to be specified (spack stack create env --compiler intel -> envs/myenv-intel; spack stack create env --compiler intel@2022 -> envs/myenv-intel-2022), which is needed to differentiate on platforms platform where more than one version of a compiler is installed.

To-do list:

• Set up script for running at least one UWM RT that runs s2swa (cpld_control_p8? should include MAPL, Scotch; see UWM testing in weekly builds #1303).
• Figure out how best to track results so we don't get in the habit of ignoring failing unit tests (Slack integration?)

Testing/implementation by platform

On each system:

• Configure ShellSetup.sh (including setting centralized build_cache dir under spack-stack/ dir)
• Set up centralized source cache
• Get it running (Jenkins, cron, GHA); test installing a few packages
• Enable notifications (email, Slack, whatever)

Platforms (check when all of the above items are complete):

• Acorn - @AlexanderRichert-NOAA - cron
• Orion - @rickgrubin-tomorrow test with Jenkins on Orion
• Nautilus - @climbfuji - either cron or Github actions

Applications affected

all

Systems affected

all (eventually)

Dependencies

none

Issue(s) addressed

#1184, #365

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[AlexanderRichert-NOAA]

closing temporarily so I don't spam the CI with jobs...

[AlexanderRichert-NOAA]

Cron entry for Acorn: 0 1 * * SUN /path/to/spack-stack-weekly-build/util/weekly_build/SpackStackBuildCache_AllSteps.sh $(date +\%y\%m\%d) /path/to/spack-stack/weekly_build acorn > /path/to/spack-stack/weekly_build/logs/cron.$(date +\%y\%m\%d).out 2>&1

[climbfuji]

@AlexanderRichert-NOAA I am pulling in the latest updates from develop once more

[rickgrubin-tomorrow]

Within a Jenkins pipeline:

Re-running everything after the recent merge, now I'm seeing a dreaded Install tree root is too long error:

[...]
17:10:48 ==> No patches needed for gmake
17:10:48 ==> gmake: Executing phase: 'install'
17:11:11 ==> Error: SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).
17:11:11   /work/noaa/epic/role-epic/jenkins/workspace/spack-stack/ufswm-test/Orion-intel/weekly/spack/user_cache/bootstrap/store/bin/sbang
[...]
17:11:11 ==> Warning: Skipping build of clingo-bootstrap-spack-wdkesgo7ubjc5jn7l2ee5s7awdugr6nr since gmake-4.4.1-e4ee4hnuu22kdvldnoenit3gsjbjaldl failed
17:11:11 ==> Warning: Skipping build of re2c-2.2-i4mprqgx2bz2zmsz3e3imasiyv5aycpc since gmake-4.4.1-e4ee4hnuu22kdvldnoenit3gsjbjaldl failed
17:11:11 ==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the following failures:
17:11:11 github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).
[...]

Ironically, the build of gmake-4.4.1 succeeded (per its build log).

Not clear on what I'm doing wrong / how to correct.

Will try taking this out of Jenkins and running in a shell (emulating a cron job).

[climbfuji]

@rickgrubin-tomorrow The error message is right there:

github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).

You need to use a shorter path.

[climbfuji]

I am fine with merging this into develop in its current stage, it works perfectly for my github action runs on nautilus. Maybe we should all just get to the step where the environments build and then populate the cache. We can make adjustments and add more capabilities in subsequent PRs, for example for downstream application testing and package unit testing.

[AlexanderRichert-NOAA]

If I can ever manage to log onto Acorn again I want to do another end-to-end test on there... But after that yes that's fine with me.

[rickgrubin-tomorrow]

@rickgrubin-tomorrow The error message is right there:

github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).

You need to use a shorter path.

I saw the message, and it's meaning was / is clear.

Curious as to why I didn't see this prior, and the only thing that changed was the merge. That said, I imagine that's not the culprit.

Thanks.

I'll have to get the pipleline to shorten:

SPACK_ENV=/work/noaa/epic/role-epic/jenkins/workspace/spack-stack/ufswm-test/Orion-intel/weekly/envs/build-unified-dev-intel

echo $SPACK_ENV | wc -c
     115