Guidance on testing CTSM on new machine for scientific validation?

[gretashum]

I'm setting up CTSM (land-only) on Hortense, which is a VSC machine at Ghent University. Since CESM prealpha tests and CTSM system tests (like clm_short) don't exactly test the correctness of results, just system functionality, and the statistical ensemble tests seem more for the full CESM, I'm wondering what I should do to validate CTSM on this new machine? I appreciated this previous post about how to approach integration testing, but I think our needs might be more basic than what's discussed. Could you give some guidance on what one should do to test the scientific consistency of CTSM on a new machine?

[samsrabin]

Tagging @briandobbins, who I think has experience here.

[briandobbins]

Unfortunately my familiarity with the land model correctness is limited; I've worked more on the atmosphere side. That said, I'd say that the best 'correctness' test we have is indeed the full ECT one for now - while it is clearly more involved than just a land-only run, it should nonetheless alert you to anything on the system that might give statistically unsound results. Eg, if it passes, then it's reasonable to assume your whole installation is working properly, inclusive of the land.

Note, however, this is only currently valid for CESM2 -- if you're porting a newer, pre-3.0 release, we don't have an ECT for that yet, though one is in the works.

Looking longer term, we are exploring using 'lower-level' statistical testing capabilities - ideally with a hierarchy, such as full model -> component model -> individual physics. It's likely, however, that we'll start with the atmosphere there, and even that's still a ways off.

If you need a hand with the ECT runs or have other questions (or ideas!), please let us know!

[gretashum]

Thanks for this response! That makes sense that we could use the full ECT, which would assess everything including the land model. Glad to hear that the lower-level statistical tests are being explored!

This might be getting too far into the weeds, but in my specific case, I was hoping to run the latest release of CTSM (5.3 or if necessary 5.2), and we currently have CESM2.2 ported to Hortense.

Since doing these tests would effectively verify the version of the land model within this release of CESM, how would you proceed? For cesm2.2.2 it looks like the land version is the release-cesm2.2.04 tag. Do you think this is similar enough to ctsm5.2(/5.3.xxx), since there wouldn't be a great way of verifying later releases of CTSM until CESM3, or do you recommend I stick with the version of CTSM that I'm able to verify is consistent? (Sorry if this a very land model-specific question!)

Alternatively, could you envision us being able to do the ECTs or other verification tests for a more recent release later on?

As a sidenote, I was also wondering if there are important updates from these instructions or updates for uploading test files for CESM2.2 since from the text it sounds like these tools are only valid for up to 2.1?

[briandobbins]

Ah, great questions -- let's start with the ECT. Basically, when porting, and comparing a specific release of CESM (no changes to the code), the ECT's value is more in verifying that that code gives statistically sound answers on your system. Put differently, we know the code is identical, so it ensures that your compilers, math libraries, operating system, etc, doesn't have anything given wrong answers. For example, in the (long) past, certain Intel compilers had a bug that gave incorrect answers, and the ECT could detect that, and people could upgrade or switch compilers to get a pass. It's also caught instances where certain optimization flags could give incorrect results, such as FMAs (fused multiply-adds), and that's another instance of just changing your compiler flags accordingly.

So, in that sense, an ECT run is still a good thing to do because, assuming you use the same compilers, libraries, flags, etc, between CESM versions, it should say, "Yes, your system environment runs CESM 2.1 correctly."

The catch here is that CESM 2.2 is an oddity, and our documentation is out of date (my fault, ironically!) -- I'd use CESM 2.1.5 for running an ECT, and given that you've done the porting for 2.2.2 already, I suspect this should be fairly straightforward. I'm happy to help if you run into any trouble whatsoever though.

As for CTSM, I'll likely defer back to @samsrabin here, but I imagine both of us would have the same first question: Is your goal to do long-running science with it, or are you in more of an exploratory phase? I ask because the 5.3 series is new and still evolving. We are moving towards a freeze for what'll be in CESM3, but not fully there yet -- so if you're doing active research, it might also depend on when you want to do production vs trial experiments. The version in CESM2.2 is approximately four years old, I believe.

So, in summary, I think you can verify your system with the ECT (using CESM 2.1), and that's likely a very strong indicator that any new releases of the code will work fine - in a sense, verifying that your port is consistent with what is expected. We'll also have an ECT for CESM3, which will include a new CTSM version, but that's still a few months out, I imagine. And there are newer versions of CTSM available now, but tuning is ongoing, so while a port can likely be trusted if an ECT passes with CESM 2.1, I wouldn't necessarily call the model validated yet. Again, I'll defer to Sam here, as he knows a lot more than I do on this.

[ekluzek]

@gretashum thanks for the conversation. A note I'm going to add to what @briandobbins has been talking about for the ECT test is that CTSM I compsets are strongly forced by the DATM forcing. As such the variability is going to be very small compared to fully coupled B compsets that the ECT test is centered around.

However you can use some of the process for the ECT testing and it would work for I compsets. So run an ensemble on the trusted machine i.e. Derecho, and then make sure your new machine gives answers within that spread.

That's something you can do for any version of the model.

[ekluzek]

I appreciated this previous post about how to approach integration testing, but I think our needs might be more basic than what's discussed.

@gretashum I'm just noting that the post there has to do with lower level testing than what you need. It has to do with validating smaller subsets of the model rather than the whole system which is what you need.

[gretashum]

Thanks so much for all these responses! This is super helpful!