Skip to content

Commit

Permalink
Regain bit-for-bit identical results between IPD and CCPP for coupled…
Browse files Browse the repository at this point in the history 
… model runs (NOAA-GFDL#86)

* make rain/snow tendency consistent with accumulated rain/snow

* put drain_cpl and dsnow_cpl in proper container

* Updates of IPD and CCPP code to regain bit-for-bit identical results for coupled model runs

* Update .gitmodules and submodule pointer for ccpp-physics for code review and testing

* gfsphysics/GFS_layer/GFS_physics_driver.F90: need to initialize local variables for bit-for-bit identical results

* Bugfix in gfsphysics/GFS_layer/GFS_typedefs.F90, allocate Tbd%drain_cpl and Tbd%dsnow_cpl when cplchm or cplflx is true

* Remove local/interstitial variables for seaice coupling, add suite definition file for S2S benchmark runs

* Rename S2S suite suite_FV3_GFS_2017_coupled_satmedmf.xml to suite_FV3_GFS_2017_satmedmf_coupled.xml

* Revert change to .gitmodules and update submodule pointer for ccpp-physics

Co-authored-by: Phil Pegion <[email protected]>
Co-authored-by: Philip Pegion <[email protected]>
  • Loading branch information
@climbfuji @pjpegion
3 people authored Mar 27, 2020
1 parent 515a83f commit 083dc6c
Show file tree
Hide file tree
Showing 8 changed files with 265 additions and 126 deletions.
2 changes: 1 addition & 1 deletion ccpp/physics
Submodule physics updated 13 files
+9 −16 physics/GFS_MP_generic.F90
+12 −9 physics/GFS_PBL_generic.F90
+8 −8 physics/GFS_PBL_generic.meta
+21 −3 physics/GFS_debug.F90
+2 −2 physics/GFS_stochastics.F90
+7 −7 physics/GFS_suite_interstitial.F90
+34 −22 physics/GFS_surface_composites.F90
+13 −24 physics/GFS_surface_generic.F90
+18 −90 physics/GFS_surface_generic.meta
+1 −1 physics/gcycle.F90
+8 −8 physics/sfc_cice.meta
+6 −6 physics/sfc_nst.f
+18 −0 physics/sfc_nst.meta
87 changes: 87 additions & 0 deletions ccpp/suites/suite_FV3_GFS_2017_satmedmf_coupled.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_GFS_2017_satmedmf_coupled" lib="ccppphys" ver="3">
<!-- <init></init> -->
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>rrtmg_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_ocean</scheme>
<scheme>lsm_noah</scheme>
<scheme>sfc_cice</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>satmedmfvdif</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>rayleigh_damp</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>zhaocarr_gscond</scheme>
<scheme>zhaocarr_precpd</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
92 changes: 92 additions & 0 deletions ccpp/suites/suite_FV3_GFS_v15p2_coupled.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?xml version="1.0" encoding="UTF-8"?>

<suite name="FV3_GFS_v15p2_coupled" lib="ccppphys" ver="3">
<!-- <init></init> -->
<group name="fast_physics">
<subcycle loop="1">
<scheme>fv_sat_adj</scheme>
</subcycle>
</group>
<group name="time_vary">
<subcycle loop="1">
<scheme>GFS_time_vary_pre</scheme>
<scheme>GFS_rrtmg_setup</scheme>
<scheme>GFS_rad_time_vary</scheme>
<scheme>GFS_phys_time_vary</scheme>
</subcycle>
</group>
<group name="radiation">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_rad_reset</scheme>
<scheme>GFS_rrtmg_pre</scheme>
<scheme>rrtmg_sw_pre</scheme>
<scheme>rrtmg_sw</scheme>
<scheme>rrtmg_sw_post</scheme>
<scheme>rrtmg_lw_pre</scheme>
<scheme>rrtmg_lw</scheme>
<scheme>rrtmg_lw_post</scheme>
<scheme>GFS_rrtmg_post</scheme>
</subcycle>
</group>
<group name="physics">
<subcycle loop="1">
<scheme>GFS_suite_interstitial_phys_reset</scheme>
<scheme>GFS_suite_stateout_reset</scheme>
<scheme>get_prs_fv3</scheme>
<scheme>GFS_suite_interstitial_1</scheme>
<scheme>GFS_surface_generic_pre</scheme>
<scheme>GFS_surface_composites_pre</scheme>
<scheme>dcyc2t3</scheme>
<scheme>GFS_surface_composites_inter</scheme>
<scheme>GFS_suite_interstitial_2</scheme>
</subcycle>
<!-- Surface iteration loop -->
<subcycle loop="2">
<scheme>sfc_diff</scheme>
<scheme>GFS_surface_loop_control_part1</scheme>
<scheme>sfc_ocean</scheme>
<scheme>lsm_noah</scheme>
<scheme>sfc_cice</scheme>
<scheme>sfc_sice</scheme>
<scheme>GFS_surface_loop_control_part2</scheme>
</subcycle>
<!-- End of surface iteration loop -->
<subcycle loop="1">
<scheme>GFS_surface_composites_post</scheme>
<scheme>sfc_diag</scheme>
<scheme>sfc_diag_post</scheme>
<scheme>GFS_surface_generic_post</scheme>
<scheme>GFS_PBL_generic_pre</scheme>
<scheme>hedmf</scheme>
<scheme>GFS_PBL_generic_post</scheme>
<scheme>GFS_GWD_generic_pre</scheme>
<scheme>cires_ugwp</scheme>
<scheme>cires_ugwp_post</scheme>
<scheme>GFS_GWD_generic_post</scheme>
<scheme>rayleigh_damp</scheme>
<scheme>GFS_suite_stateout_update</scheme>
<scheme>ozphys_2015</scheme>
<scheme>h2ophys</scheme>
<scheme>GFS_DCNV_generic_pre</scheme>
<scheme>get_phi_fv3</scheme>
<scheme>GFS_suite_interstitial_3</scheme>
<scheme>samfdeepcnv</scheme>
<scheme>GFS_DCNV_generic_post</scheme>
<scheme>GFS_SCNV_generic_pre</scheme>
<scheme>samfshalcnv</scheme>
<scheme>GFS_SCNV_generic_post</scheme>
<scheme>GFS_suite_interstitial_4</scheme>
<scheme>cnvc90</scheme>
<scheme>GFS_MP_generic_pre</scheme>
<scheme>gfdl_cloud_microphys</scheme>
<scheme>GFS_MP_generic_post</scheme>
<scheme>maximum_hourly_diagnostics</scheme>
</subcycle>
</group>
<group name="stochastics">
<subcycle loop="1">
<scheme>GFS_stochastics</scheme>
</subcycle>
</group>
<!-- <finalize></finalize> -->
</suite>
93 changes: 43 additions & 50 deletions gfsphysics/GFS_layer/GFS_physics_driver.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -544,9 +544,6 @@ subroutine GFS_physics_driver &
doms, psautco_l, prautco_l, ocalnirbm_cpl, ocalnirdf_cpl, &
ocalvisbm_cpl, ocalvisdf_cpl, dtzm, temrain1, t2mmp, q2mp, &
psaur_l, praur_l, &
!--- coupling inputs for physics
dtsfc_cice, dqsfc_cice, dusfc_cice, dvsfc_cice, &
! dtsfc_cice, dqsfc_cice, dusfc_cice, dvsfc_cice, ulwsfc_cice, &
!--- for CS-convection
wcbmax

Expand Down Expand Up @@ -678,16 +675,15 @@ subroutine GFS_physics_driver &
real :: pshltr,QCQ,rh02
real(kind=kind_phys), allocatable, dimension(:,:) :: den

!! Initialize local variables (mainly for debugging purposes, because the
!! corresponding variables Interstitial(nt)%... are reset to zero every time);
!! these variables are only modified over parts of the entire domain (related
!! to land surface mask etc.)
!! Initialize local variables (for debugging purposes only,
!! because the corresponding variables Interstitial(nt)%...
!! are reset to zero every time).
!snowmt = 0.
!gamq = 0.
!gamt = 0.
!gflx = 0.
!hflx = 0.
!

!! Strictly speaking, this is not required. But when
!! hunting for bit-for-bit differences, doing the same as
!! in GFS_suite_stateout_reset makes life a lot easier.
Expand Down Expand Up @@ -911,10 +907,6 @@ subroutine GFS_physics_driver &
! --- set initial quantities for stochastic physics deltas
if (Model%do_sppt) then
Tbd%dtdtr = zero
do i=1,im
Tbd%drain_cpl(i) = Coupling%rain_cpl (i)
Tbd%dsnow_cpl(i) = Coupling%snow_cpl (i)
enddo
endif

! mg, sfc-perts
Expand Down Expand Up @@ -1113,14 +1105,6 @@ subroutine GFS_physics_driver &
do i=1,im
islmsk_cice(i) = nint(Coupling%slimskin_cpl(i))
flag_cice(i) = (islmsk_cice(i) == 4)

if (flag_cice(i)) then
! ulwsfc_cice(i) = Coupling%ulwsfcin_cpl(i)
dusfc_cice(i) = Coupling%dusfcin_cpl(i)
dvsfc_cice(i) = Coupling%dvsfcin_cpl(i)
dtsfc_cice(i) = Coupling%dtsfcin_cpl(i)
dqsfc_cice(i) = Coupling%dqsfcin_cpl(i)
endif
enddo
endif
!*## CCPP ##
Expand All @@ -1145,9 +1129,8 @@ subroutine GFS_physics_driver &
endif
endif
if (fice(i) < one) then
wet(i)=.true. !some open ocean/lake water exists
wet(i)=.true. ! some open ocean/lake water exists
if (.not. Model%cplflx) Sfcprop%tsfco(i) = max(Sfcprop%tsfco(i), Sfcprop%tisfc(i), tgice)

end if
else
fice(i) = zero
Expand Down Expand Up @@ -1659,13 +1642,14 @@ subroutine GFS_physics_driver &
sbsno(i) = zero
snowc(i) = zero
snohf(i) = zero
!## CCPP ##* GFS_surface_generic.F90/GFS_surface_generic_pre_run
Diag%zlvl(i) = Statein%phil(i,1) * onebg
Diag%smcwlt2(i) = zero
Diag%smcref2(i) = zero

wind(i) = max(sqrt(Statein%ugrs(i,1)*Statein%ugrs(i,1) + &
Statein%vgrs(i,1)*Statein%vgrs(i,1)) &
+ max(zero, min(Tbd%phy_f2d(i,Model%num_p2d), 30.0)), one)
!*## CCPP ##
enddo
!*## CCPP ##

Expand Down Expand Up @@ -1909,6 +1893,30 @@ subroutine GFS_physics_driver &

endif !lsm

!! Strictly speaking, this is not required. But when
!! hunting for bit-for-bit differences, updating the
!! subsurface variables in the Sfcprop DDT makes
!! life a lot easier
!if (Model%frac_grid) then
! do k=1,lsoil
! do i=1,im
! if (dry(i)) then
! Sfcprop%smc(i,k) = smsoil(i,k)
! Sfcprop%stc(i,k) = stsoil(i,k)
! Sfcprop%slc(i,k) = slsoil(i,k)
! endif
! enddo
! enddo
!else
! do k=1,lsoil
! do i=1,im
! Sfcprop%smc(i,k) = smsoil(i,k)
! Sfcprop%stc(i,k) = stsoil(i,k)
! Sfcprop%slc(i,k) = slsoil(i,k)
! enddo
! enddo
!endif

! if (lprnt) write(0,*)' tseabeficemodel =',Sfcprop%tsfc(ipr),' me=',me &
! &, ' kdt=',kdt,' tsfc32=',tsfc3(ipr,2),' fice=',fice(ipr) &
! &,' stsoil=',stsoil(ipr,:)
Expand All @@ -1931,8 +1939,9 @@ subroutine GFS_physics_driver &
(im, Statein%tgrs(:,1), &
Statein%qgrs(:,1,1), cd3(:,2), cdq3(:,2), &
Statein%prsl(:,1), wind, &
flag_cice, flag_iter, dqsfc_cice, dtsfc_cice, &
dusfc_cice, dvsfc_cice, &
flag_cice, flag_iter, &
Coupling%dqsfcin_cpl, Coupling%dtsfcin_cpl, &
Coupling%dusfcin_cpl, Coupling%dvsfcin_cpl, &
! --- outputs:
qss3(:,2), cmm3(:,2), chh3(:,2), evap3(:,2), hflx3(:,2), &
stress3(:,2))
Expand Down Expand Up @@ -2103,15 +2112,11 @@ subroutine GFS_physics_driver &
ep1d(i) = ep1d3(i,k)
Sfcprop%weasd(i) = weasd3(i,k)
Sfcprop%snowd(i) = snowd3(i,k)

evap(i) = evap3(i,k)
hflx(i) = hflx3(i,k)
qss(i) = qss3(i,k)
Sfcprop%tsfc(i) = tsfc3(i,k)

Diag%cmm(i) = cmm3(i,k)
Diag%chh(i) = chh3(i,k)

Sfcprop%zorll(i) = zorl3(i,1)
Sfcprop%zorlo(i) = zorl3(i,3)

Expand All @@ -2120,7 +2125,6 @@ subroutine GFS_physics_driver &
txo = one - txi
evap(i) = txi * evap3(i,2) + txo * evap3(i,3)
hflx(i) = txi * hflx3(i,2) + txo * hflx3(i,3)
! Sfcprop%tsfc(i) = txi * tice(i) + txo * tsfc3(i,3)
Sfcprop%tsfc(i) = txi * tsfc3(i,2) + txo * tsfc3(i,3)
else ! return updated lake ice thickness & concentration to global array
if (islmsk(i) == 2) then
Expand Down Expand Up @@ -2839,10 +2843,10 @@ subroutine GFS_physics_driver &
do i=1,im
if (Sfcprop%oceanfrac(i) > zero) then ! Ocean only, NO LAKES
if (Sfcprop%fice(i) > one - epsln) then ! no open water, thus use results from CICE
Coupling%dusfci_cpl(i) = dusfc_cice(i)
Coupling%dvsfci_cpl(i) = dvsfc_cice(i)
Coupling%dtsfci_cpl(i) = dtsfc_cice(i)
Coupling%dqsfci_cpl(i) = dqsfc_cice(i)
Coupling%dusfci_cpl(i) = Coupling%dusfcin_cpl(i)
Coupling%dvsfci_cpl(i) = Coupling%dvsfcin_cpl(i)
Coupling%dtsfci_cpl(i) = Coupling%dtsfcin_cpl(i)
Coupling%dqsfci_cpl(i) = Coupling%dqsfcin_cpl(i)
elseif (icy(i) .or. dry(i)) then ! use stress_ocean from sfc_diff for opw component at mixed point
tem1 = max(Diag%q1(i), 1.e-8)
rho = Statein%prsl(i,1) / (con_rd*Diag%t1(i)*(one+con_fvirt*tem1))
Expand All @@ -2856,8 +2860,6 @@ subroutine GFS_physics_driver &
endif
Coupling%dtsfci_cpl(i) = con_cp * rho * hflx3(i,3) ! sensible heat flux over open ocean
Coupling%dqsfci_cpl(i) = con_hvap * rho * evap3(i,3) ! latent heat flux over open ocean
! if (lprnt .and. i == ipr) write(0,*)' hflx33=',hflx3(i,3),' evap33=',evap3(i,3), &
! ' con_cp=',con_cp,' rho=',rho,' con_hvap=',con_hvap
else ! use results from PBL scheme for 100% open ocean
Coupling%dusfci_cpl(i) = dusfc1(i)
Coupling%dvsfci_cpl(i) = dvsfc1(i)
Expand Down Expand Up @@ -5283,7 +5285,7 @@ subroutine GFS_physics_driver &
!*## CCPP ##
!## CCPP ##* GFS_MP_generic.F90/GFS_MP_generic_post_run
Diag%rain(:) = Diag%rainc(:) + frain * rain1(:) ! total rain per timestep

! --- get the amount of different precip type for Noah MP
! --- convert from m/dtp to mm/s
if (Model%lsm==Model%lsm_noahmp) then
Expand Down Expand Up @@ -5463,10 +5465,10 @@ subroutine GFS_physics_driver &

if (Model%cplflx .or. Model%cplchm) then
do i = 1, im
Coupling%rain_cpl(i) = Coupling%rain_cpl(i) &
+ Diag%rain(i) * (one-Sfcprop%srflag(i))
Coupling%snow_cpl(i) = Coupling%snow_cpl(i) &
+ Diag%rain(i) * Sfcprop%srflag(i)
Tbd%drain_cpl(i)= Diag%rain(i) * (one-Sfcprop%srflag(i))
Tbd%dsnow_cpl(i)= Diag%rain(i) * Sfcprop%srflag(i)
Coupling%rain_cpl(i) = Coupling%rain_cpl(i) + Tbd%drain_cpl(i)
Coupling%snow_cpl(i) = Coupling%snow_cpl(i) + Tbd%dsnow_cpl(i)
enddo
endif

Expand Down Expand Up @@ -5557,15 +5559,6 @@ subroutine GFS_physics_driver &
if (Model%do_sppt) then
!--- radiation heating rate
Tbd%dtdtr(1:im,:) = Tbd%dtdtr(1:im,:) + dtdtc(1:im,:)*dtf
do i = 1, im
if (t850(i) > 273.16) then
!--- change in change in rain precip
Tbd%drain_cpl(i) = Diag%rain(i) - Tbd%drain_cpl(i)
else
!--- change in change in snow precip
Tbd%dsnow_cpl(i) = Diag%rain(i) - Tbd%dsnow_cpl(i)
endif
enddo
endif
!*## CCPP ##
!## CCPP ##* This block is not in the CCPP since it is not needed in the CCPP.
Expand Down
Loading

0 comments on commit 083dc6c

Please sign in to comment.