CTSM on Derecho

[negin513]

Hello Team, 👋

I hope you are doing fine!
Some users have reached out to us, asking about running CTSM on Derecho.
They are particularly interested in knowing which tags they should use for this purpose. Could you please let me know if there are any Derecho-compatible tags available at the moment?

Looking forward to your response.

[ekluzek]

Hello @negin513!

Unfortunately we do not currently have a CTSM tag that works out of the box on Derecho. We have an issue where we've identified the things that we need to do to get it working #1995

What I would recommend that users do is to use one of the latest CESM tags. Personally I like to start with the last CESM beta tag, as it's had the most testing. So right now that would be...

https://github.com/ESCOMP/CESM/tree/cesm2_3_beta15

The very latest alpha tag is

https://github.com/ESCOMP/cesm/tree/cesm2_3_alpha16d

and it does have a few machine updates for Derecho.

That should work for running normal CTSM cases on Derecho. However, they might need to set the PE layout to something that works or is more optimized for Derecho. So the issue above might help give them some advice about that.

Hopefully, that helps. Let us know if there's more questions or concerns. And feel free to send them our way with CTSM issues.

Take care...

[negin513]

Hello @ekluzek ,

Thanks so much for your response. Yes, to fully utilize full Derecho nodes, the PE layout needs to be modified.

I will point them to those CESM tags.

Have a great day!
Negin

[ekluzek]

Hello @negin513!

We now have a CTSM tag that works has working PE layouts and is tested on Derecho for CTSM:

ctsm5.1.dev158

(the previous tag ctsm5.1.dev157 also has the externals update as the minimum needed to run on Derecho.

Thanks for keeping reminding us about the need to move to Derecho. I can't say we are early, but at least it's happening.

Take care...

[negin513]

Thanks @ekluzek for the update. Glad to hear about the tag with working PE layouts and I will let users who ask this questions know about this. :-)

Have a great day!
Negin

[samsrabin]

For now, it should be possible to build CTSM on Derecho by manually updating Externals.cfg to match the version on the CESM3_dev branch: https://github.com/ESCOMP/CTSM/blob/CESM3_dev/Externals.cfg

[samsrabin]

For a 1-degree I case:, I set up a similar-to-Cheyenne processor layout like so:

./xmlchange NTASKS_CPL=1716,NTASKS_ATM=36,NTASKS_LND=1716,NTASKS_ICE=1716,NTASKS_OCN=1716,NTASKS_ROF=1716,NTASKS_GLC=1716,NTASKS_WAV=1716,NTASKS_ESP=1
./case.setup --reset

This provides DATM with the same number of processors as before for a 1-degree I case, and it ensures that the land+DATM processors sum to a multiple of 128 (the number of processors per node on Derecho).

This has not been optimized, but it does at least work!

[ekluzek]

This is awesome @samsrabin. Can you look at your timing and just tell us how fast DATM and CTSM are each running at and the total throughput rate?

[samsrabin]

• DATM: ~365 model years per day
• CTSM: ~105
• Total: ~73

[wwieder]

see #2284