Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize wavepostpnt #2657

Merged
merged 16 commits into from
Jun 17, 2024
Merged

Optimize wavepostpnt #2657

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
d87a550
Optimize the wavepostpnt scripts
DavidHuber-NOAA Jun 3, 2024
76a02aa
Merge remote-tracking branch 'origin/develop' into feature/opt_pnt
DavidHuber-NOAA Jun 4, 2024
709e6dc
Further optimize the wavepostpnt job
DavidHuber-NOAA Jun 4, 2024
0acaa37
Add double quotes around variables
DavidHuber-NOAA Jun 4, 2024
db300eb
Apply suggestions from code review
DavidHuber-NOAA Jun 4, 2024
8f4f7ab
Pregenerate sed-searchable paths.
DavidHuber-NOAA Jun 5, 2024
fbf73b5
Merge remote-tracking branch 'origin/develop' into feature/opt_pnt
DavidHuber-NOAA Jun 7, 2024
670a7ef
Revert buggy optimization for finding IBP points
DavidHuber-NOAA Jun 7, 2024
2d94656
Escape '&' in sed commands
DavidHuber-NOAA Jun 7, 2024
0b4a6da
Remove double-quotes from wave_mpmd
DavidHuber-NOAA Jun 7, 2024
1e49ab9
Increase wallclock for wavepostbndpnt jobs
DavidHuber-NOAA Jun 10, 2024
a7522cf
Adjust data requirement for wavepostbndpntbll job for experiment fore…
DavidHuber-NOAA Jun 10, 2024
6dfedca
Key wavepostbndpntbll log file off of FHMAX_WAV_IBP
DavidHuber-NOAA Jun 10, 2024
9817f16
Correct FHMAX_WAV_IBP cap hours
DavidHuber-NOAA Jun 11, 2024
2e9953b
Set last_fhr off of FHMAX_WAV_IBP
DavidHuber-NOAA Jun 11, 2024
8f40ef7
Merge branch 'develop' into feature/opt_pnt
DavidHuber-NOAA Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parm/config/gefs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ case ${step} in
# The wavepost*pnt* jobs are I/O heavy and do not scale well to large nodes.
# Limit the number of tasks/node to 40.
"wavepostbndpnt")
export wtime_wavepostbndpnt="01:00:00"
export wtime_wavepostbndpnt="03:00:00"
export npe_wavepostbndpnt=240
export nth_wavepostbndpnt=1
export npe_node_wavepostbndpnt=$(( npe_node_max / nth_wavepostbndpnt ))
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gefs/config.wave
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ export WAVEWND_FID=
# The start time reflects the number of hindcast hours prior to the cycle initial time
export WAVHINDH=0
export FHMAX_WAV_IBP=180
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_GFS} ; fi
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_WAV} ; fi

# gridded and point output rate
export DTFLD_WAV=$(( FHOUT_HF_WAV * 3600 ))
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ case ${step} in
# The wavepost*pnt* jobs are I/O heavy and do not scale well to large nodes.
# Limit the number of tasks/node to 40.
"wavepostbndpnt")
export wtime_wavepostbndpnt="01:00:00"
export wtime_wavepostbndpnt="03:00:00"
export npe_wavepostbndpnt=240
export nth_wavepostbndpnt=1
export npe_node_wavepostbndpnt=$(( npe_node_max / nth_wavepostbndpnt ))
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.wave
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ if [[ "${RUN}" == "gfs" ]]; then
fi
export WAVHINDH=0
export FHMAX_WAV_IBP=180
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_GFS} ; fi
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_WAV} ; fi

# gridded and point output rate
export DTFLD_WAV=$(( FHOUT_HF_WAV * 3600 ))
Expand Down
60 changes: 29 additions & 31 deletions scripts/exgfs_wave_post_pnt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,14 +289,14 @@ source "${USHgfs}/preamble.sh"
fi

# Create new buoy_log.ww3
cat buoy.loc | awk '{print $3}' | sed 's/'\''//g' > ibp_tags
awk '{print $3}' buoy.loc | sed 's/'\''//g' > ibp_tags
grep -F -f ibp_tags buoy_log.ww3 > buoy_log.tmp
rm -f buoy_log.dat
mv buoy_log.tmp buoy_log.dat

grep -F -f ibp_tags buoy_lst.loc > buoy_tmp1.loc
#sed '$d' buoy_tmp1.loc > buoy_tmp2.loc
buoys=$(awk '{ print $1 }' buoy_tmp1.loc)
awk '{ print $1 }' buoy_tmp1.loc > buoy_lst.txt
Nb=$(wc buoy_tmp1.loc | awk '{ print $1 }')
rm -f buoy_tmp1.loc

Expand Down Expand Up @@ -349,6 +349,8 @@ source "${USHgfs}/preamble.sh"

# 1.a.2 Loop over forecast time to generate post files
fhr=$FHMIN_WAV
# Generated sed-searchable paths
escaped_USHgfs="${USHgfs//\//\\\/}"
while [ $fhr -le $FHMAX_WAV_PNT ]; do

echo " Creating the wave point scripts at : $(date)"
Expand All @@ -365,6 +367,7 @@ source "${USHgfs}/preamble.sh"

# Create instances of directories for spec and gridded output
export SPECDATA=${DATA}/output_$YMDHMS
escaped_SPECDATA="${SPECDATA//\//\\\/}"
export BULLDATA=${DATA}/output_$YMDHMS
cp $DATA/mod_def.${waveuoutpGRD} mod_def.${waveuoutpGRD}

Expand All @@ -385,19 +388,15 @@ source "${USHgfs}/preamble.sh"
if [ "$DOSPC_WAV" = 'YES' ]
then
export dtspec=3600.
for buoy in $buoys
do
echo "${USHgfs}/wave_outp_spec.sh $buoy $ymdh spec $SPECDATA > $SPECDATA/spec_$buoy.out 2>&1" >> tmpcmdfile.$FH3
done
# Construct the wave_outp_spec (spec) command to run on each buoy in buoy_lst.txt
sed "s/^\(.*\)$/${escaped_USHgfs}\/wave_outp_spec.sh \1 ${ymdh} spec ${escaped_SPECDATA} > ${escaped_SPECDATA}\/spec_\1.out 2>\&1/" buoy_lst.txt >> "tmpcmdfile.${FH3}"
fi

if [ "$DOBLL_WAV" = 'YES' ]
then
export dtspec=3600.
for buoy in $buoys
do
echo "${USHgfs}/wave_outp_spec.sh $buoy $ymdh bull $SPECDATA > $SPECDATA/bull_$buoy.out 2>&1" >> tmpcmdfile.$FH3
done
# Construct the wave_outp_spec (bull) command to run on each buoy in buoy_lst.txt
sed "s/^\(.*\)$/${escaped_USHgfs}\/wave_outp_spec.sh \1 ${ymdh} bull ${escaped_SPECDATA} > ${escaped_SPECDATA}\/bull_\1.out 2>\&1/" buoy_lst.txt >> "tmpcmdfile.${FH3}"
fi

split -n l/1/10 tmpcmdfile.$FH3 > cmdfile.${FH3}.01
Expand Down Expand Up @@ -503,46 +502,43 @@ source "${USHgfs}/preamble.sh"

cd $DATA

echo "Before create cmdfile for cat bouy : $(date)"
rm -f cmdfile.bouy
touch cmdfile.bouy
chmod 744 cmdfile.bouy
echo "Before create cmdfile for cat buoy : $(date)"
rm -f cmdfile.buoy
touch cmdfile.buoy
chmod 744 cmdfile.buoy
CATOUTDIR=${DATA}/pnt_cat_out
escaped_CATOUTDIR="${CATOUTDIR//\//\\\/}"
mkdir -p ${CATOUTDIR}

if [ "$DOSPC_WAV" = 'YES' ]
then
for buoy in $buoys
do
echo "${USHgfs}/wave_outp_cat.sh $buoy $FHMAX_WAV_PNT spec > ${CATOUTDIR}/spec_cat_$buoy.out 2>&1" >> cmdfile.bouy
done
# Construct wave_outp_cat (spec) call for each buoy in buoy_lst.txt
sed "s/^\(.*\)$/${escaped_USHgfs}\/wave_outp_cat.sh \1 ${FHMAX_WAV_PNT} spec > ${escaped_CATOUTDIR}\/spec_cat_\1.out 2>\&1/" buoy_lst.txt >> cmdfile.buoy
fi

if [ "$DOBLL_WAV" = 'YES' ]
then
for buoy in $buoys
do
echo "${USHgfs}/wave_outp_cat.sh $buoy $FHMAX_WAV_PNT bull > ${CATOUTDIR}/bull_cat_$buoy.out 2>&1" >> cmdfile.bouy
done
# Construct wave_outp_cat (bull) call for each buoy in buoy_lst.txt
sed "s/^\(.*\)$/${escaped_USHgfs}\/wave_outp_cat.sh \1 ${FHMAX_WAV_PNT} bull > ${escaped_CATOUTDIR}\/bull_cat_\1.out 2>\&1/" buoy_lst.txt >> cmdfile.buoy
fi

if [ ${CFP_MP:-"NO"} = "YES" ]; then
nfile=0
ifile=0
iline=1
ifirst='yes'
nlines=$( wc -l cmdfile.bouy | awk '{print $1}' )
nlines=$( wc -l < cmdfile.buoy)
while [ $iline -le $nlines ]; do
line=$( sed -n ''$iline'p' cmdfile.bouy )
line=$( sed -n ''$iline'p' cmdfile.buoy )
if [ -z "$line" ]; then
break
else
if [ "$ifirst" = 'yes' ]; then
echo "#!/bin/sh" > cmdfile.bouy.$nfile
echo "$nfile cmdfile.bouy.$nfile" >> cmdmprogbouy
chmod 744 cmdfile.bouy.$nfile
echo "#!/bin/sh" > cmdfile.buoy.$nfile
echo "$nfile cmdfile.buoy.$nfile" >> cmdmprogbuoy
chmod 744 cmdfile.buoy.$nfile
fi
echo $line >> cmdfile.bouy.$nfile
echo $line >> cmdfile.buoy.$nfile
nfile=$(( nfile + 1 ))
if [ $nfile -eq $NTASKS ]; then
nfile=0
Expand All @@ -553,7 +549,7 @@ source "${USHgfs}/preamble.sh"
done
fi

wavenproc=$(wc -l cmdfile.bouy | awk '{print $1}')
wavenproc=$(wc -l < cmdfile.buoy)
wavenproc=$(echo $((${wavenproc}<${NTASKS}?${wavenproc}:${NTASKS})))

set +x
Expand All @@ -566,9 +562,11 @@ source "${USHgfs}/preamble.sh"
if [ "$wavenproc" -gt '1' ]
then
if [ ${CFP_MP:-"NO"} = "YES" ]; then
${wavempexec} -n ${wavenproc} ${wave_mpmd} cmdmprogbouy
# shellcheck disable=SC2086
${wavempexec} -n "${wavenproc}" ${wave_mpmd} cmdmprogbuoy
else
${wavempexec} ${wavenproc} ${wave_mpmd} cmdfile.bouy
# shellcheck disable=SC2086
${wavempexec} "${wavenproc}" ${wave_mpmd} cmdfile.buoy
fi
exit=$?
else
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
39 changes: 15 additions & 24 deletions ush/wave_outp_spec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,21 +73,7 @@ source "${USHgfs}/preamble.sh"
exit 1
else
buoy=$bloc
grep $buoy ${DATA}/buoy_log.ww3 > tmp_list.loc
while read line
do
buoy_name=$(echo $line | awk '{print $2}')
if [ $buoy = $buoy_name ]
then
point=$(echo $line | awk '{ print $1 }')
set +x
echo " Location ID/# : $buoy (${point})"
echo " Spectral output start time : $ymdh "
echo ' '
set_trace
break
fi
done < tmp_list.loc
point=$(awk "{if (\$2 == \"${buoy}\"){print \$1; exit} }" "${DATA}/buoy_log.ww3")
if [ -z "$point" ]
then
set +x
Expand All @@ -97,6 +83,11 @@ source "${USHgfs}/preamble.sh"
echo ' '
set_trace
exit 2
else
set +x
echo " Location ID/# : $buoy (${point})"
echo " Spectral output start time : $ymdh "
echo ' '
fi
fi

Expand Down Expand Up @@ -200,27 +191,27 @@ source "${USHgfs}/preamble.sh"
then
if [ "$specdir" = "bull" ]
then
cat $outfile | sed -e '9,$d' >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.bull
cat $coutfile | sed -e '8,$d' >> ${STA_DIR}/c${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.cbull
sed '9,$d' "${outfile}" >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.bull"
sed '8,$d' "${coutfile}" >> "${STA_DIR}/c${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.cbull"
else
cat $outfile >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.spec
cat $outfile >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.spec"
fi
elif [ "${ymdh}" = "${YMDHE}" ]
then
if [ "$specdir" = "bull" ]
then
cat $outfile | sed -e '1,7d' >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.bull
cat $coutfile | sed -e '1,6d' >> ${STA_DIR}/c${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.cbull
sed '1,7d' "${outfile}" >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.bull"
sed '1,6d' "${coutfile}" >> "${STA_DIR}/c${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.cbull"
else
cat $outfile | sed -n "/^${YMD} ${HMS}$/,\$p" >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.spec
sed -n "/^${YMD} ${HMS}$/,\$p" "${outfile}" >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.spec"
fi
else
if [ "$specdir" = "bull" ]
then
cat $outfile | sed -e '1,7d' | sed -e '2,$d' >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.bull
cat $coutfile | sed -e '1,6d' | sed -e '2,$d' >> ${STA_DIR}/c${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.cbull
sed '8q;d' "${outfile}" >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.bull"
sed '7q;d' "${coutfile}" >> "${STA_DIR}/c${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.cbull"
else
cat $outfile | sed -n "/^${YMD} ${HMS}$/,\$p" >> ${STA_DIR}/${specdir}fhr/$WAV_MOD_TAG.${ymdh}.$buoy.spec
sed -n "/^${YMD} ${HMS}$/,\$p" "${outfile}" >> "${STA_DIR}/${specdir}fhr/${WAV_MOD_TAG}.${ymdh}.${buoy}.spec"
fi
fi
else
Expand Down
14 changes: 12 additions & 2 deletions workflow/rocoto/gefs_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,8 +386,18 @@ def wavepostbndpnt(self):
def wavepostbndpntbll(self):
deps = []
atmos_hist_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'})
# Is there any reason this is 180?
data = f'{atmos_hist_path}/{self.cdump}[email protected]'

# The wavepostbndpntbll job runs on forecast hours up to FHMAX_WAV_IBP
wave_cfg = self._configs['wave']
last_fhr = Tasks._get_forecast_hours(self.cdump,
wave_cfg,
component='atmos')[-1]

fhmax_wav_ibp = wave_cfg['FHMAX_WAV_IBP']
if last_fhr > fhmax_wav_ibp:
last_fhr = fhmax_wav_ibp
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved

data = f'{atmos_hist_path}/{self.cdump}[email protected]{last_fhr:03d}.txt'
dep_dict = {'type': 'data', 'data': data}
deps.append(rocoto.add_dependency(dep_dict))

Expand Down
13 changes: 12 additions & 1 deletion workflow/rocoto/gfs_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1175,9 +1175,20 @@ def wavepostbndpnt(self):
return task

def wavepostbndpntbll(self):

# The wavepostbndpntbll job runs on forecast hours up to FHMAX_WAV_IBP
wave_cfg = self._configs['wave']
last_fhr = Tasks._get_forecast_hours(self.cdump,
wave_cfg,
component='atmos')[-1]

fhmax_wav_ibp = wave_cfg['FHMAX_WAV_IBP']
if last_fhr > fhmax_wav_ibp:
last_fhr = fhmax_wav_ibp
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved

deps = []
atmos_hist_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"])
data = f'{atmos_hist_path}/{self.cdump}[email protected].logf180.txt'
data = f'{atmos_hist_path}/{self.cdump}[email protected].logf{last_fhr:03d}.txt'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WalterKolczynski-NOAA FHMAX_GFS and FHMAX_WAV_IBP can differ if FHMAX_GFS is greater than 180 hours.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JessicaMeixner-NOAA It seems like FHMAX_WAV_IBP should default to FHMAX_WAV instead of FHMAX_GFS in

global-workflow/parm/config/gfs/config.wave

Lines 100 to 105 in e7909af

if [[ "${RUN}" == "gfs" ]]; then
export FHMAX_WAV=${FHMAX_WAV_GFS}
fi
export WAVHINDH=0
export FHMAX_WAV_IBP=180
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_GFS} ; fi

And thus change to

if [[ "${RUN}" == "gfs" ]]; then
  export FHMAX_WAV=${FHMAX_WAV_GFS}
fi
export WAVHINDH=0
export FHMAX_WAV_IBP=180
if (( FHMAX_WAV < FHMAX_WAV_IBP )); then export FHMAX_WAV_IBP=${FHMAX_WAV} ; fi

Do you think that is correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DavidHuber-NOAA nice catch, yes I do think this is correct.

dep_dict = {'type': 'data', 'data': data}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)
Expand Down
Loading