ruamel for gmao-swell-env?

[Dooruk]

In our templating and writing dictionaries to files, we noticed that the order of output YAMLs change and they are written alphabetically:

https://github.com/GEOS-ESM/swell/blob/96dd41c8a548b61e51334bda8477d3e481e964df/src/swell/tasks/run_jedi_variational_executable.py#L119C1-L122C89

This worked for us so far, however Ricardo noticed the scrambling of UFO filters might be causing answers to differ between GSI and SWELL, so we are considering multiple options to preserve order. I tested with a simple conversion to OrderedDict which works in a temporary way. I noticed EWOK uses NamedDict and ruamel package, which might be a more robust fix. So I was wondering if ruamel could be added to the gmao-swell-env and/or if it has too many dependencies?

https://github.com/JCSDA-internal/ewok/blob/d26a9f805e6068ab637a9004ba07084e8188cb02/src/yamltools/template.py#L18

More information here GEOS-ESM/swell#409 (comment)

[climbfuji]

Since we build the unified environment on Discover, can you simply load the existing module for py-ruamel for the time being? It's available after you load stack-COMPILER, stack-MPI, stack-PYTHON.

[Dooruk]

Hmm, I don't see py-ruamel in my module list after loading via (or on SCU16):

module purge
module use /discover/swdev/gmao_SIteam/modulefiles-SLES15
module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles
module load ecflow/5.11.4

module use /gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-1.7.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
module load stack-intel/2021.10.0
module load stack-intel-oneapi-mpi/2021.10.0
module load stack-python/3.10.13

[climbfuji]

Just to confirm, I have py-ruamel-yaml and py-ruamel-yaml-clib in my list of available modules. Thse are not what you are looking for?

[Dooruk]

Just to confirm, I have py-ruamel-yaml and py-ruamel-yaml-clib in my list of available modules. Thse are not what you are looking for?

Do you see those with module list after loading the same modules I listed above?

[climbfuji]

I can't log into discover at the moment, but I am checking the unified environment on my local machine (it is supposed to be identical)

[climbfuji]

@srherbener or @ashley314 Since I can't log into Discover at the moment (or not at all anymore?), can one of you check please if the modules py-ruamel-yaml and py-ruamel-yaml-clib are installed in the existing (1.7.0) and new (1.8.0) unified environments on Discover? Thanks very much!

[ashley314]

@climbfuji I see both of those modules on discover in spack-stack 1.7.0 and 1.8.0

[climbfuji]

Thanks @ashley314 - @Dooruk can you check again, please?

[Dooruk]

@climbfuji I see both of those modules on discover in spack-stack 1.7.0 and 1.8.0

Thanks for checking this @ashley314. Could you test the following script that loads and tests ruamel with the following on SCU17 please? I'm getting ModuleNotFoundError: No module named 'ruamel'.

module purge
module use /discover/swdev/gmao_SIteam/modulefiles-SLES15
module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles
module load ecflow/5.11.4

module use /gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-1.7.0/envs/ue-intel-2021.10.0/install/modulefiles/Core
module load stack-intel/2021.10.0
module load stack-intel-oneapi-mpi/2021.10.0
module load stack-python/3.10.13
module load jedi-fv3-env
module load soca-env
module load gmao-swell-env/1.0.0

#Test importing ruamel
python3 -c "from ruamel.yaml import YAML"

[Dooruk]

Thanks @ashley314 - @Dooruk can you check again, please?

I must be missing an env load.

[climbfuji]

Please load the pu-ruamel-yaml etc modules explicitly after loading your other modules. It may be that the ruamel modules are part of ewok-env, but as I said in my comment #1253 (comment) from last week, please load those py-ruamel modules directly.

[Dooruk]

It's there, my bad, I thought it should already be loaded not in the available modules!

[climbfuji]

Sorry for not being clear. We should still add the modules you want (which ones exactly) to gmao-swell-env for spack-stack-1.9.0, but until then you'll have to keep loading them manually. (I'll keep this issue open until we added the correct modules to gmao-swell-env).

[Dooruk]

Sounds good, thank you.