What is the CTSM Philosophy for CESM prealpha and prebeta tests?

[ekluzek]

CTSM manages the tests for CESM prealpha, prebeta and aux_cime_baselines tests in our testlist -- what is the philosophy for what tests should be in that list?

The current list has largely been maintained moving forward as is. So it would be good to revisit what's in the list. Some guidance we've had here are the following:

1. We wanted these tests to be a subset of the aux_clm we run for tags (so our aux_clm testing would usually catch problems before CESM testing)
2. We want prealpha/prebeta tests to be I case versions of B and F test cases that are run in CESM and CAM
3. Since submodules are updated more frequently in CESM tags -- we wanted tests to catch problems with submodule updates
4. We wanted to test a good sampling of important configurations for CTSM with different: physics versions, important options, resolutions, compilers, and DEBUG on and off
5. We wanted to test the most important CTSM configurations while leaving the more obscure options for our own testing
6. The aux_cime_baseline tests are to be the important standard tests to evaluate answer changes in CESM testing

In general the prealpha test list is run more often so it should be smaller, and with lower resolutions and shorter simulations. The prebeta should also exercise the standard resolution used for coupled simulations (right now ne30pg3). The aux_cime_baseline testlist is run nightly as a standard part of CESM testing. As such it should be a fairly short testlist.

Examples of "4" is the MKSURFDATAESMF test. Examples of "2" are the PTSMODE tests and fire-emission, dry-deposition tests that ensure that SCAM and CAM-Chem tests in CAM and CESM are more likely to continue working.

I think the above principles make sense to me and we should retain most of this. We can probably reduce some of the tests in "5" and "6".and let the aux_clm testlist take care of more of the obscure testing.

We certainly should reevaluate "2" and make sure we are covering CESM and CAM tests for B and F testcases in the I case framework for CTSM. We should also evaluate if anything else is needed for "4" we do want to maintain mpi-serial tests here for example. And I wonder if we should for example add the FUNITCTSM test for prebeta.