Add task to process reforecast variables to save on WCOSS2 #2680
Commits on May 20, 2024
-
Add task to extract reforecast variables
A rocoto task has been added to process the reforecast variables. A config file has been created for this task. Resources specific to this task have also been added to config.resources.
Commits on May 21, 2024
-
Add top level script for extractvars task
The topmost extractvars script has been added to the global workflow rocoto subdirectory.
Commits on May 22, 2024
Commits on May 28, 2024
-
Update extractvars job dependency
The extractvars job dependency has been updated so that extractvars is triggered by the completion of wave_post_grid.
Commits on May 29, 2024
-
Add JGLOBAL script for extractvars task
A new JGLOBAL script has been added for the extractvars task.
-
Add scripts to process atmos reforecast variables
Scripts have been added which allow the processing of atmosphere reforecast variables.
-
Add parm files for extractvars task
Parm files have been added that provide a variable list specific to reforecast variables to be saved on disk.
-
The extractvars scripts have been simplified.
Commits on May 31, 2024
-
Add script to process ocean and ice
A script has been added to process the ocean and ice variables for the reforecast. Parm files for ocean and ice have also been added.
Commits on Jun 3, 2024
-
A script has been added to process the wave variables for the reforecast.
-
The script that processes waves in the extractvars task has been cleaned up.
-
Add wave forecast frequency variable
A wave output frequency variable has been added that allows a specific wave forecast output frequency for data saved on disk.
Commits on Jun 4, 2024
-
Shellnorms is set to run on every git push.
-
An FHMAX variable has been added to the wave script in the extractvars task so that an unbound error can be avoided.
-
Clean up JGLOBAL_EXTRACTVARS script
The JGLOBAL_EXTRACTVARS script has been refined.
-
The extactvars script has been cleaned up.
-
-
-
-
-
Commits on Jun 5, 2024
-
-
Simplify ocnice extractvars script
Unneeded lines have been removed from the ocnice extractvars script.
-
An unused variable ensname has been removed from the ocnice extractvars script.
-
Clean up wave script in extractvars task
The wave script in the extractvars task has been cleaned up.
-
Add more fixes to wave script in extractvars
More fixes have been added to the wave script in the extractvars task.
-
Define COM directories for extractvars task
COM directories have been defined for the extractvars task. Data generated in the extractvars task has the ability to be copied to COMROOT.
Commits on Jun 7, 2024
-
Create an option to remove extractvars task
An option in config.base has been created that allows the user to include or not include the extractvars task in the workflow.
Commits on Jun 10, 2024
-
-
Modify wave parm file for extractvars task
The wave parm file for the extractvars task has been modify to meet stakeholder and resource requirements.
Commits on Jun 11, 2024
Commits on Jun 12, 2024
-
Improve task dependency for extractvars
The dependencies have been modified for the extractvars task.
-
An unused variable has been removed form the wave script in extractvars.
-
Add DO_EXTRACTVARS to default yaml
The variable DO_EXTRACTVARS has been added to the gefs default yaml.
-
The config.base for GEFS has been modified to consider a default DO_EXTRACTVARS in the default yaml file.
-
Fix dependencies for extractvars
The metatask dependencies have been corrected for the extractvars task in gefs_tasks.py.
-
The FHOUT_OCNICE variable has been updated to FHOUT_OCN and FHOUT_ICE.
-
Remove unused variables from config.extractvars
Unused variables have been removed from config.extractvars. A couple of new variables have been added.
-
More unused variables have been removed from config.extractvars.
-
A nonexistant variable has been removed from exglobal_extractvars.
-
Modify scripts to process netcdf input
The extractvars scripts have been modified so that there is a capability for netcdf ocn and ice input to be processed.
-
-
Commits on Jun 14, 2024
-
Add checks for ocean/ice variable extraction
Checks have been added to ensure all requested ocean and ice variables are available in the input netcdf files.
-
Commits on Jun 18, 2024
-
Add checks to the extractvars atmos scripts
Some checks have been added to the extractvars atmos scripts. A new script has also been created that contains some functions used to process the atmos variables for the extractvars task.
-
Commits on Jun 21, 2024
-
Apply initial fix for ocean filename mismatch
An initial fix has been added so that the workflow can properly be link the MOM6 native files when there is an offset.
Commits on Jun 23, 2024
-
Revert "Apply initial fix for ocean filename mismatch"
This reverts commit 5774c2a.
Commits on Jun 24, 2024
-
Apply an improved fix for ocean filename mismatch
A new diag_table has been created that is specfically used for when OFFSET_START_HOUR is greater than 0. Using this diag_table will allow for the ocean filenames to properly match to what is created in the workflow.
-
Commits on Jun 25, 2024
-
Increase flexibility of extractvars dependencies
A conditional statement has been added to allow for certain dependencies to be omitted from extractvars for components that are not set to run.
-
Allow only specific components to be processed
Only components that are set to run will be processed in the extractvars task.
Commits on Jul 2, 2024
-
-
-
Move extractvars variable defs to config
The extractvars variables have been moved from exscript to config.extractvars.
Commits on Jul 8, 2024
-
Replace COM with COMIN and COMOUT
The COM variables have been replaced with COMIN and COMOUT in the extractvars task.
-
Revert changes related to offsets
Changes that are related to offsets have been removed.
Commits on Jul 9, 2024
-
JGLOBAL_EXTRACTVARS has been renamed to JGEFS_EXTRACTVARS.
-
Add comment to explain DO_EXTRACTVARS
A brief description has been added for DO_EXTRACTVARS.
-
Polish some if-statements in extractvars
Some if-statements have been refined in extractvars.
-
Revert "Rename JGLOBAL_EXTRACTVARS"
This reverts commit c7f8364.
-
Simplify com var defs in gefs_atmos_extractvars
The defining of the com var varialbes have been refined in gefs_atmos_extractvars
-
Rename oufile in gefs_atmos_extractvars
The oufile variable has been renamed to outfile in gefs_atmos_extractvars.
-
Add varlist as argument for check_atmos
The varlist variable has been added as an argument for the check_atmos function.
-
A variable has been added in config.extractvars that defines a compression utility.
-
Rename oufile1 in gefs_ocnice_extractvars
The oufile1 variable has been renamed to outfile in gefs_ocnice_extractvars.
-
Rename oufile1 in gefs_wav_extractvars
The oufile1 variable in gefs_wav_extractvars has been renamed to outfile.
-
Remove mode arugment from mkdir
The mode argument has been removed from mkdir in JGLOBAL_EXTRACTVARS.
-
Remove FHMAX and use FHMAX_GFS
The FHMAX variable has been removed from config.extractvars. FHMAX_GFS from config.base is used in the extractvars task instead of FHMAX.
-
Remove duplicate variable in config.extractvars
A duplicate variable ocnres has been removed from config.extractvars.
-
Remove FHMAXHF, FHMAXLF and FHMAXHF
FHMAXHF, FHMAXLF and FHMAXHF have been removed from config.extractvars and have been replaced with FHMAX_HF_GFS, FHOUT_GFS and FHMAX_HF_GFS from config.base, respectively.
-
Check that FHMAX_GFS is greater than FHMAX_HF_GFS
An error check has been added to make sure FHMAX_GFS is greater than FHMAX_HF_GFS. A fatal error will occur if this check fails.
-
Moved parmfile variables to config.extractvars
The parmfile variables have been moved from exglobal_extractvars to config.extractvars. Some of the parmfile descriptions have also been corrected.
-
Add extractvars function to copy output to COMROOT
A function has been added that checks if an output file exits. If the output file exits, then the output file is copied from RUNDIRS to COMROOT.
-
Copy ocn, ice and wave files individually
The ocn, ice and wave extractvar scripts have been modified so that the output can be copied to COMROOT individually.
-
Split COMOUT RFCST ATMOS into two directories
The COMOUT RFCST ATMOS variable has been split into two directories. There is one directory for f2d variables and one directory for f3d variables.
-
-
Replace gefs string with RUN variable
The string "gefs" has been replaced with the variable RUN.
Commits on Jul 10, 2024
-
-
Replace while loops with for loops where possible
Loops that have a constant interval have been changed from while loops to for loops.
-
-
Modify extractvars dependencies
The extractvars dependencies have been modified. When the prod jobs for a certain member are finished, the extractvars task for that member will be triggered. Double quotes have also been added to the atmos extractvars script.
-
Remove gefs string from ush script filenames
The string "gefs" has been removed from the extractvars ush script filenames.
-
Update resources used in extractvars
The resources requested have been reduced in config.resources.
Commits on Jul 11, 2024
-
Use more efficient method to generate parmlist
A more efficient method is used to generate a partial parmlist in extractvars.
-
Add job-specific variable for FHMAX_HF_GFS
A job-specific variable for FHMAX_HF_GFS has been added called FHMAX_HF_EV. FHMAX_HF_GFS has been replaced by FHMAX_HF_EV accordingly.
-
-
Move temporary atmos files into subdata directory
The temporary atmos files for the extractvars task have been moved to the subdata directory.
-
Rename varlist in check_atmos function
The varlist variable in the check_atmos function has been renamed to varlistl.
-
Address shell check error and fix error
A shell check has been addressed. Also, an empty partial parm file is created before any wgrib2 functions are called in the atmos ush extractvars script. This prevents any no such file or directory error when wgrib2 is used.
-
Add more detail to warning message
More detail has been added to a warning message when a requested parm variable cannot be found in any of the input atmos grib2 files for a given lead time and given member.
-
Commits on Jul 16, 2024
-
Declare functions and source in exscript
Extractvars functions have been declared in extractvars_tools. The extractvars functions are sourced in exglobal_extractvars instead of atmos_extractvars.
-
The command cp has been replaced with the variable NCP in extractvars_tools.
-
-
Remove source extractvars_tools from ush scripts
The source extractvars_tools.sh has been removed from the ush scripts. Source extractvars_tools has been moved to the exscript.
-
Rename wav_extractvars to wave_extractvars
The wav_extractvars script has been renamed to wave_extractvars.
-
Pass dcnt into daily_avg_atmos and add description
The dcnt variable has been passed into the daily_avg_atmos function. A brief description of dcnt, vcnt and fcnt have also been added.
-
Replace ${cycle} with t${cyc}z
The variable ${cycle} has been replaced with t${cyc}z. -
Pass outres into daily_avg_atmos function
The outres variable is passed into daily_avg_atmos function.
-
Simplify the declaring of COM variables
The COM variables for ocean, ice and waves are declared all in one command.
-
-
Add check for FHOUT_WAV_EXTRACT
A check has been added to make sure FHOUT_WAV is a factor of FHOUT_WAV_EXTRACT. If it is not, FHOUT_WAV_EXTRACT is set to FHOUT_WAV. Also added a space to some comments.
-
Remove an unneccesary WGRIB2 definition from exglobal_extractvars.
-
-
Use variable subdata in ocnice/wave ush scripts
The variable subdata replaces outdirpre and DATA in the ocnice and wave ush scripts.
-
Remove unneccesary lines in exglobal_extractvars
The mkdir portions have been removed from exglobal_extractvars. Also, "mem${ENSMEM}_" has been removed from directory names.
Commits on Jul 17, 2024
-
Remove gen_parmlist and loop through infiles
The gen_parmlist function has been removed and a for-loop has been created to loop through infile1 and inflile2. The variable check all occurs in the check_atmos function.
-
Loop through infile when creating davg files
A for-loop has been created to loop through the infiles when calculating the daily-averaged files.
-
Address shellcheck error and add comment
A comment has been added to explain why an error should not be returned in check_atmos function.
-
Add file availability check in atmos script
A file availability check has been added for the daily-averaged data file extraction.
-
Add option to extract levels for ocean 3d vars
An option has been added to extract a range of levels from the 3d variables in the netcdf ocean products.
-
-
-
-
Grib2 support has been removed from the extractvars task.
-
Add fatal error if data file does not exist
A fatal error will occur if an output data file does not exist.
Commits on Jul 18, 2024
-
Rename FHMAX_HF_EV to FHMAX_HF_GFS
The job-specific variable FHMAX_HF_EV has been renamed to FHMAX_HF_GFS according to a reviewer's suggestion.
-
Change output directory to ARCDIR
The output directory for extractvars has been moved from COMOUT to ARCDIR.
-
Move mkdirs for arcdir to J-JOB script
The arcdir directories are being created in the JGLOBAL_EXTRACTVARS instead of config.extractvars. Also, directories for a specific component will only be made if that component is set to run.
-
Add brief documentation to scripts and functions
Brief documentation has been added to the extractvars scripts and functions.
-
Modify description for daily_atmos
A portion of the description for daily_atmos has been removed since what was mentioned is not a requirement for the input grib2 file.
-
Revert "Modify description for daily_atmos"
This reverts commit bf19e74.
-
Modify and correct description for daily_atmos
The correct portion of the description for daily_atmos has been removed since what was mentioned is not a requirement for the input grib2 file. Another portion that was incorrectly removed has been reinstated.
