Add capability to run forecast in segments #2795
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalterKolczynski-NOAA
requested review from
aerorahul,
KateFriedman-NOAA,
EricSinsky-NOAA,
DavidHuber-NOAA and
NeilBarton-NOAA
|
I still have some more testing to do (and I think documentation to update), but wanted to get a draft out since people are going on leave. |
WalterKolczynski-NOAA
force-pushed
the
feature/fcst_segments
branch
from
7cd1192
to
cd3d7c8
Compare
|
Will check and see if the documentation needs to be updated Friday, but I think the last few bugs from this update are gone. |
WalterKolczynski-NOAA
added
the
CI-Hercules-Ready
emcbot
added
CI-Hercules-Building
|
CI Passed Hercules at |
|
Hercules testing completed successfully, resetting label. |
DavidHuber-NOAA
removed
the
CI-Hercules-Running
WalterKolczynski-NOAA
force-pushed
the
feature/fcst_segments
branch
from
cd3d7c8
to
eb17358
Compare
|
Appears no documentation updates are needed at this time after all. |
aerorahul
reviewed
Jul 29, 2024
parm/config/gfs/config.base
Outdated
Comment on lines
294
to
304
| export FCST_SEGMENTS_STR_GFS="@FCST_SEGMENTS_GFS@" | ||
| IFS=', ' read -ra FCST_SEGMENTS_GFS <<< "${FCST_SEGMENTS_STR_GFS}" | ||
| if (( ${FCST_SEGMENT:- -1} < 0 )); then | ||
| # Jobs other than the forecast don't care about segments, only the | ||
| # absolute start and end | ||
| declare -x FHMIN_GFS=${FCST_SEGMENTS_GFS[0]} | ||
| declare -x FHMAX_GFS=${FCST_SEGMENTS_GFS[-1]} | ||
| else | ||
| declare -x FHMIN_GFS=${FCST_SEGMENTS_GFS[${FCST_SEGMENT}]} | ||
| declare -x FHMAX_GFS=${FCST_SEGMENTS_GFS[${FCST_SEGMENT}+1]} | ||
| fi |
There was a problem hiding this comment.
Is there a way to not do any of this in a config file and calculate this in a j-job or exscript?
There was a problem hiding this comment.
Not without massive additional changes. FHMAX_GFS especially gets used later in this config, and then also in the job-specific configs that would be sourced immediately afterwards.
There was a problem hiding this comment.
Re-done how we discussed Mon afternoon.
WalterKolczynski-NOAA
added
CI-Hercules-Ready
emcbot
added
CI-Hercules-Building
WalterKolczynski-NOAA
force-pushed
the
feature/fcst_segments
branch
from
d9f01cb
to
6816722
Compare
emcbot
added
the
CI-Hercules-Passed
WalterKolczynski-NOAA
force-pushed
the
feature/fcst_segments
branch
from
3877cc5
to
c73eecd
Compare
WalterKolczynski-NOAA
added
CI-Wcoss2-Ready
emcbot
added
CI-Wcoss2-Building
|
emcbot
added
CI-Wcoss2-Running
|
Automated global-workflow Testing Results: |
emcbot
added
CI-Wcoss2-Failed
|
Experiment C96C48_hybatmDA_c73eecd2 FAIL on Wcoss2 at 08/10/24 07:06:33 AM Error logs: Follow link here to view the contents of the above file(s): (link) |
WalterKolczynski-NOAA
added
CI-Wcoss2-Ready
emcbot
added
CI-Wcoss2-Building
|
emcbot
added
CI-Wcoss2-Running
|
Automated global-workflow Testing Results: |
emcbot
added
CI-Wcoss2-Passed
|
All CI Test Cases Passed on Wcoss2: |
| @@ -24,7 +24,7 @@ if [[ "${DOIAU}" == "YES" ]]; then | |||
| export aero_bkg_times="3,6,9" | |||
| export JEDIYAML="${PARMgfs}/gdas/aero/variational/3dvar_fgat_gfs_aero.yaml.j2" | |||
| else | |||
| export aero_bkg_times="6" | |||
| export aero_bkg_times="6," # Trailing comma is necessary so this is treated as a list | |||
DavidHuber-NOAA
approved these changes
Aug 12, 2024
DavidHuber-NOAA
added a commit
to DavidHuber-NOAA/global-workflow
that referenced
this pull request
Aug 13, 2024
…e_rocoto * origin/develop: Jenkins Pipeline Updates (NOAA-EMC#2815) Add Gaea C5 to CI (NOAA-EMC#2814) Add support for forecast-only runs on AWS (NOAA-EMC#2711) Add fixes to products for when REPLAY IC's are used (NOAA-EMC#2755) Add capability to run forecast in segments (NOAA-EMC#2795)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds the ability to run a forecast in segments instead of all at once. To accomplish this, a new local
checkpntsvariable is introduced toconfig.baseto contain a comma-separated list of intermediate stopping points for the forecast. This is combined withFHMIN_GFSandFHMAX_GFSto create a comma-separated stringFCST_SEGMENTSwith all the start/end points that is used byconfig.fcstand rocoto workflow. Capability to parse these into python lists was added to wxflow in an accompanying PR. Ifcheckpntsis an empty string, this will result in a single-segment forecast.To accommodate the new segment metatasks that must be run serially, the capability of
create_task()was expanded to allow a dictionary key ofis_serial, which controls whether a metatask is parallel or serial using pre-existing capability in rocoto. The default when not given is parallel (i.e. most metatasks).Resolves #2274
Refs NOAA-EMC/wxflow#39
Refs NOAA-EMC/wxflow#40
Type of change
Change characteristics
How has this been tested?
Checklist