Change spack workflow for end-user to simplify their experience

[atteggiani]

From previous discussions related to access-hive.org.au/#777.

Description

Currently, the spack workflow for an end-user, on Gadi, requires setting up spack.
For this, they are required to:

• Clone three repositories (spack, spack-config, spack-packages)
• Create symlinks of the spack configurations

In addition, before they can build an executable, they will have to set configuration parameters (env variables and paths) by sourcing the spack-config/spack-enable.bash (this step is required for any new session).

All these steps add complexity, especially for a beginner user that might only want to re-build a model with a change in the source code of one component (e.g.: they developed a new convection scheme for the UM, they want to use a custom version of CABLE (embedded in UM), they have changed some physics in MOM, etc.).

Ideal workflow

The optimal spack build workflow on Gadi, for an end-user to produce a custom-build of an ACCESS-NRI-released model is listed below.
Note: These are the minimum steps required to build a model, but any user could customize their build in a more "advanced" way if they want to (for example create custom environments, custom spack-packages, etc.).
Also, each step is representative of what happens from the end-user point of view, and might not exactly reflect all that happens under the hood.

1. Load a module to access the right spack executable

   module load access-spack
   

2. (ONLY IF NEEDED) Run a command that configures spack with custom configuration (one-off step)

   spack-config [--dir <path-to-releases-parent-dir] [--any-other-important-flag]
   

3. (ONLY IF NEEDED) Load a pre-configured environment for ACCESS-NRI-released models, depending on the model they want to build.

   spack env load access-esm1.5
   

4. Build the executables (in any session)
   Proceed with the actual build of the executables.

   spack install ... 
   

I think overall this change would highly enhance the user experience and increase spack usability for ACCESS model builds, especially among the beginner users.

[harshula]

Apparently Gadi's version of modules may be too old to achieve the desired outcome.

[atteggiani]

To better try and address this issue, we might need more details on on why an old(er) version of modules would prevent the realisation of the desired workflow.

@aidanheerdegen would you be able to elaborate more on it?

[aidanheerdegen]

@aidanheerdegen would you be able to elaborate more on it?

There is a topic on the forum with many replies that contains information about attempts to do this

https://forum.access-hive.org.au/t/shared-access-spack-configuration/227

[atteggiani]

Thank you for the useful forum post link @aidan, and for already trying to set up something on the lines of what I proposed in this issue.

Reading the forum post I could capture the following issues that need to be addressed for the proposed workflow to be realised:

• Gadi's version of modules does not allow sourcing a file within the modulefiles and still being able to revert the shell to it’s previous state when unloading the module.
• The system Python version has a bug with the lustre file system, so the SPACK_PYTHON environment variable needs to be set to a known good python3 executable.
• The SPACK_INSTALL_TREE environment variable (that defines where packages will be installed by Spack) needs to be unique for every user ad potentially "customisable" by a user, to decide where they want to build their "custom" packages. It is still preferred this variable to be set when the spack module is loaded.

Is there anything I missed or do you think there are any other issues that need to be addressed?

[harshula]

Upstream PR: spack/spack#11871

[atteggiani]

That PR is useful, but too long and most of those things are not really needed for the workflow mentioned above.
The solution to not being able to modify system environments would be "cloning" a system environment and modify that.

Also, it has been sitting there for very long (> 5 years) and I don't see it being merged anytime soon, so I would suggest finding our best approach to our needs.

Reading the forum post I could capture the following issues that need to be addressed for the proposed workflow to be realised:

• Gadi's version of modules does not allow sourcing a file within the modulefiles and still being able to revert the shell to it’s previous state when unloading the module.
• The system Python version has a bug with the lustre file system, so the SPACK_PYTHON environment variable needs to be set to a known good python3 executable.
• The SPACK_INSTALL_TREE environment variable (that defines where packages will be installed by Spack) needs to be unique for every user ad potentially "customisable" by a user, to decide where they want to build their "custom" packages. It is still preferred this variable to be set when the spack module is loaded.

I have solutions for all these points above (need to be tested but they should work).
Are there any other requirements we need to consider?