Enable cycling atmosphere DA in 3DEnVar mode with coupled ensemble. #1718

aerorahul · 2023-06-28T21:23:50Z

Description
This PR:

allows self cycling of the ocean and ice components in a cycled atmosphere with 3DEnVar (without IAU).
allows staging of initial conditions for coupled ensemble when setting up cycled ensemble.

This PR also includes:

numerous improvements involving appropriate locations for defining relevant variables for components.

This PR does not:

address archiving of ensemble data when cycling
probably many other issues that are necessary e.g. IAU

Partially addresses #613

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

3 cycles with 3DEnVar atmosphere and coupled ensemble
C48 S2S forecast was also tested. Waves and Aerosols have not been tested (again).

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

aerorahul · 2023-06-28T21:25:32Z

I will add more details to the description later.

HenryRWinterbottom · 2023-06-28T21:28:04Z

parm/config/gfs/config.base.emc.dyn

@@ -167,7 +167,7 @@ case "${CASE}" in
        export waveGRD='glo_500'
        ;;
    "C96")
-        export OCNRES=100
+        export OCNRES=500


Why not support OCNRES=100 as well?

It's supported, it's just not a default for any ATM resolution.

OK, thank you for the clarification @WalterKolczynski-NOAA.

We need to revisit this section in config.base as more combinations are possible, but the limitation here is the bash configuration.

HenryRWinterbottom · 2023-06-28T21:28:34Z

parm/config/gfs/config.efcs


 # TODO: Possibly need OCNRES_ENKF, ICERES_ENKF, WAVRES_ENKF too
 if [[ ${DO_OCN} == "YES" ]]; then
  case "${CASE_ENS}" in
    "C48") export OCNRES=500;;
-    "C96") export OCNRES=100;;
+    "C96") export OCNRES=500;;


See https://github.com/NOAA-EMC/global-workflow/pull/1718/files#r1245789899.

This is set because the deterministic and the ensemble share the same ocean resolution, unlike the atmosphere where we have dual resolutions. I am sure we can support dual resolution ocean as well, but its not worth the hassle at this moment.

HenryRWinterbottom · 2023-06-28T21:31:23Z

I haven't be requested as a reviewer, but other than the OCNRES=100 questions this looks good to me FWIW.

guillaumevernieres

Exiting stuff! Thanks @aerorahul , looks perfect to me.
I did not test your new feature

HenryRWinterbottom

LGTM. Approved.

WalterKolczynski-NOAA

Generally looks good.

scripts/exgdas_enkf_fcst.sh

ush/forecast_postdet.sh

ush/forecast_predet.sh

ush/parsing_namelists_FV3.sh

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

github-advanced-security

shellcheck found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

ush/forecast_postdet.sh

ush/parsing_namelists_FV3.sh

emcbot · 2023-06-29T22:25:52Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Jun 29 21:12:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 21:16:40 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 22:25:39 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:44 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:47 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:50 UTC 2023 for experiment C96_atm3DVar

emcbot · 2023-06-29T23:12:15Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Jun 29 21:12:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 21:16:40 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 22:25:39 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:44 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:47 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:50 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 23:12:14 UTC 2023
with 18 successfully completed jobs

emcbot · 2023-06-29T23:12:17Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Jun 29 16:36:24 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 16:38:36 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 18:11:57 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:05 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:09 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:15 CDT 2023 for experiment C96C48_hybatmDA

emcbot · 2023-06-29T23:57:07Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Jun 29 16:36:24 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 16:38:36 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 18:11:57 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:05 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:09 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:15 CDT 2023 for experiment C96C48_hybatmDA
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 18:57:06 CDT 2023
with 18 successfully completed jobs

emcbot · 2023-06-30T00:42:13Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Jun 29 16:36:24 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 16:38:36 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 18:11:57 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:05 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:09 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:15 CDT 2023 for experiment C96C48_hybatmDA
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 18:57:06 CDT 2023
with 18 successfully completed jobs
Experiment C96_atm3DVar Terminated: *FAILED*
Experiment C96_atm3DVar Terminated with 1 tasks failed at Thu Jun 29 19:42:09 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1718/RUNTESTS/C96_atm3DVar/COMROT/C96_atm3DVar/logs/2021122100/gdasvrfy.log

emcbot · 2023-06-30T00:57:15Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Jun 29 21:12:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 21:16:40 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 22:25:39 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:44 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:47 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:50 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 23:12:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Fri Jun 30 00:57:15 UTC 2023
with 148 successfully completed jobs

aerorahul · 2023-06-30T01:10:12Z

Automated global-workflow Testing Results:

Machine: Orion
Start: Thu Jun 29 16:36:24 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 16:38:36 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 18:11:57 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:05 CDT 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:09 CDT 2023 for experiment C96_atm3DVar
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 18:12:15 CDT 2023 for experiment C96C48_hybatmDA
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 18:57:06 CDT 2023
with 18 successfully completed jobs
Experiment C96_atm3DVar Terminated: *FAILED*
Experiment C96_atm3DVar Terminated with 1 tasks failed at Thu Jun 29 19:42:09 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1718/RUNTESTS/C96_atm3DVar/COMROT/C96_atm3DVar/logs/2021122100/gdasvrfy.log

The error is:

mv: cannot move ‘bad_pen.2021122100’ to a subdirectory of itself, ‘/work/noaa/global/mterry/monitor/radmon/stats/C96_atm3DVar/gdas.20211221/00/./bad_pen.2021122100’

@EdwardSafford-NOAA
We are seeing random failures in the GSI monitoring jobs like the one here. I experienced similar failures in one of our previous CI runs. Would you be able to test the GSI monitoring scripts and address these so our CI system can continue testing?

aerorahul · 2023-06-30T01:13:45Z

@TerrenceMcGuinness-NOAA @WalterKolczynski-NOAA
I see what is happening in the failure on Orion.
Data is being copied to a common directory: /work/noaa/global/mterry/monitor/radmon/stats
as seen in the failure log: /work2/noaa/stmp/GFS_CI_ROOT/PR/1718/RUNTESTS/C96_atm3DVar/COMROT/C96_atm3DVar/logs/2021122100/gdasvrfy.log
Note how this directory is not PR specific. All tests/experiments will be writing to this space, which is why this job and some others will start to fail when several PRs are being tested as part of the CI.
We need to have flexibility in defining this path for CI test.

emcbot · 2023-06-30T01:51:11Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Jun 29 21:12:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 21:16:40 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 22:25:39 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:44 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:47 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:50 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 23:12:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Fri Jun 30 00:57:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Fri Jun 30 01:51:11 UTC 2023
with 86 successfully completed jobs

emcbot · 2023-06-30T01:54:12Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Jun 29 21:12:22 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Thu Jun 29 21:16:40 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Thu Jun 29 22:25:39 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:44 UTC 2023 for experiment C48_S2S
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:47 UTC 2023 for experiment C96C48_hybatmDA
Created experiment:            *SUCCESS*
Case setup: Completed at Thu Jun 29 22:25:50 UTC 2023 for experiment C96_atm3DVar
Experiment C48_S2S completed: *SUCCESS*
Experiment C48_S2S Completed at Thu Jun 29 23:12:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA completed: *SUCCESS*
Experiment C96C48_hybatmDA Completed at Fri Jun 30 00:57:15 UTC 2023
with 148 successfully completed jobs
Experiment C96_atm3DVar completed: *SUCCESS*
Experiment C96_atm3DVar Completed at Fri Jun 30 01:51:11 UTC 2023
with 86 successfully completed jobs

aerorahul · 2023-06-30T02:58:03Z

@NeilBarton-NOAA
This has passed on Hera. The failure on Orion is related to a gdasvrfy issue not a result of changes in this PR.
Please review at your convenience as we would like to enable IAU and 4DEnVar subsequently and build on for GEFS forecasts (and reforecast) capabilities.
Thanks!

addressed reviewer comments. requesting re-review.

NeilBarton-NOAA · 2023-06-30T12:28:54Z

This looks good to me. An important step. I don't have time to run it myself

EdwardSafford-NOAA · 2023-06-30T15:22:21Z

@aerorahul I've reviewed the CI failure and understand that the issue is that the location for RadMon output location is not specific to the PR/PR# when run as a CI test. This is also true for the other DA monitors, which define the output location as $NOSCRUB/monitor/[monitor name]. $NOSCRUB is not uniquely defined by PR for CI tests. Do you wish me to undertake that change for global-workflow?

aerorahul added 5 commits May 4, 2023 11:36

update setup_expt.py to link ocean/ice ics for ensemble members

8c2a515

Merge branch 'develop' into feature/setup_ens_ics

5e8e1f6

wip

c6d3935

Merge branch 'develop' into feature/setup_ens_ics

bab5b67

cycled for 3 cycles with coupled ensemble

2c80256

aerorahul requested review from WalterKolczynski-NOAA, NeilBarton-NOAA and guillaumevernieres June 28, 2023 21:25

HenryRWinterbottom reviewed Jun 28, 2023

View reviewed changes

aerorahul requested a review from HenryRWinterbottom June 28, 2023 21:38

guillaumevernieres approved these changes Jun 28, 2023

View reviewed changes

HenryRWinterbottom approved these changes Jun 28, 2023

View reviewed changes

WalterKolczynski-NOAA previously requested changes Jun 28, 2023

View reviewed changes

aerorahul and others added 6 commits June 28, 2023 21:55

Apply suggestions from code review

eb57158

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/forecast_postdet.sh

10503ec

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/forecast_postdet.sh

4243202

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/forecast_postdet.sh

4a6b0c1

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Update ush/parsing_namelists_FV3.sh

711a088

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Merge branch 'develop' into feature/setup_ens_ics

cd73e26

github-advanced-security bot found potential problems Jun 29, 2023

View reviewed changes

aerorahul added 2 commits June 29, 2023 15:02

address shellcheck errors

5398408

more shellcheck findings

ae821b7

github-advanced-security bot found potential problems Jun 29, 2023

View reviewed changes

aerorahul added 3 commits June 29, 2023 15:33

some more shellcheck fixes

725fe84

more fixes

a41c0aa

some more of the fixes

69d8f80

aerorahul requested a review from WalterKolczynski-NOAA June 29, 2023 21:09

TerrenceMcGuinness-NOAA added the CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion label Jun 29, 2023

emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Jun 29, 2023

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Jun 30, 2023

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jun 30, 2023

WalterKolczynski-NOAA approved these changes Jun 30, 2023

View reviewed changes

NeilBarton-NOAA approved these changes Jun 30, 2023

View reviewed changes

Merge branch 'develop' into feature/setup_ens_ics

f645ab4

WalterKolczynski-NOAA merged commit 188344a into NOAA-EMC:develop Jun 30, 2023
8 checks passed

aerorahul deleted the feature/setup_ens_ics branch June 30, 2023 16:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable cycling atmosphere DA in 3DEnVar mode with coupled ensemble. #1718

Enable cycling atmosphere DA in 3DEnVar mode with coupled ensemble. #1718

aerorahul commented Jun 28, 2023

aerorahul commented Jun 28, 2023

HenryRWinterbottom Jun 28, 2023

WalterKolczynski-NOAA Jun 28, 2023

HenryRWinterbottom Jun 28, 2023

aerorahul Jun 29, 2023

HenryRWinterbottom Jun 28, 2023

aerorahul Jun 29, 2023

HenryRWinterbottom commented Jun 28, 2023

guillaumevernieres left a comment

HenryRWinterbottom left a comment

WalterKolczynski-NOAA left a comment

github-advanced-security bot left a comment

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 30, 2023

emcbot commented Jun 30, 2023

aerorahul commented Jun 30, 2023

aerorahul commented Jun 30, 2023

emcbot commented Jun 30, 2023

emcbot commented Jun 30, 2023

aerorahul commented Jun 30, 2023

NeilBarton-NOAA commented Jun 30, 2023

EdwardSafford-NOAA commented Jun 30, 2023

Enable cycling atmosphere DA in 3DEnVar mode with coupled ensemble. #1718

Enable cycling atmosphere DA in 3DEnVar mode with coupled ensemble. #1718

Conversation

aerorahul commented Jun 28, 2023

aerorahul commented Jun 28, 2023

HenryRWinterbottom Jun 28, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jun 28, 2023

Choose a reason for hiding this comment

HenryRWinterbottom Jun 28, 2023

Choose a reason for hiding this comment

aerorahul Jun 29, 2023

Choose a reason for hiding this comment

HenryRWinterbottom Jun 28, 2023

Choose a reason for hiding this comment

aerorahul Jun 29, 2023

Choose a reason for hiding this comment

HenryRWinterbottom commented Jun 28, 2023

guillaumevernieres left a comment

Choose a reason for hiding this comment

HenryRWinterbottom left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

github-advanced-security bot left a comment

Choose a reason for hiding this comment

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 29, 2023

emcbot commented Jun 30, 2023

emcbot commented Jun 30, 2023

aerorahul commented Jun 30, 2023

aerorahul commented Jun 30, 2023

emcbot commented Jun 30, 2023

emcbot commented Jun 30, 2023

aerorahul commented Jun 30, 2023

NeilBarton-NOAA commented Jun 30, 2023

EdwardSafford-NOAA commented Jun 30, 2023