The PR for this issue is just about done. But one last thing I would like to do is unify the number of cores/tasks used for the FV3-JEDI and MPAS-JEDI ctests so that we can more accurately compare the runtime. For FV3-JEDI it seems like the cores need to match the layout settings for the model, so it would be easier to change the MPAS-JEDI settings.

For MPAS-JEDI, we need a matching graphinfo file for the number of cores used to run JEDI. We have conus12km_mpas.graph.info.part.120 and conus12km_mpas.graph.info.part.180 but none for 160 cores that are used for FV3-JEDI.

Does anyone know how to generate a conus12km_mpas.graph.info.part.160? @guoqing-noaa, @chunhuazhou, @TingLei-NOAA?

@SamuelDegelia-NOAA I think @xyzemc had created those files for my case, who might also show you how to create those info files . Also, changing the mpi layout means you also need to change bump files. (that will be interesting to see if different mpi layout would produce identical results).

@SamuelDegelia-NOAA I think @xyzemc had created those files for my case, who might also show you how to create those info files . Also, changing the mpi layout means you also need to change bump files. (that will be interesting to see if different mpi layout would produce identical results).

Thank you @TingLei-NOAA for the reminder, I'll also regenerate the bumploc files. I agree that it will be interesting to see if that changes the results.

@SamuelDegelia-NOAA I think @xyzemc had created those files for my case, who might also show you how to create those info files . Also, changing the mpi layout means you also need to change bump files. (that will be interesting to see if different mpi layout would produce identical results).

Here is what I did before:

ln -sf /scratch2/BMC/wrfruc/mpas/MPAS-DATA/metis-5.1.0-gcc9.1.0/bin/gpmetis .
gpmetis -minconn -contig -niter=200 CONUS.graph.info 36

Thanks @xyzemc, that worked! I was also missing CONUS.graph.info (called conus12km_mpas.graph.info for the new 12-km case), but I was able to recreate it following @chunhuazhou's guide in #41. I added this and the new conus12km_mpas.graph.info.part.160 to the staged data.

@TingLei-NOAA Here is the impact of changing the number of MPI tasks for our MPAS-JEDI EnVar test case. The bumploc files were regenerated for the plot with 160 tasks. There are some very very small differences so the results are not identical, but overall the impact is negligible. Also, the run with 160 tasks ran about 40 seconds faster.

MPI tasks = 120:

MPI tasks = 160:

@SamuelDegelia-NOAA Thanks for sharing! Sure, it is difficult to produce identical results for procedures like bump (and mgbf). But good to know the differences are very small for bump.

A couple of updates on this issue before the PR regarding unifying the localization radii between the GSI and JEDI test cases. I have disabled SDL/VDL in the GSI test case to better match the configurations we use in MPAS-JEDI and FV3-JEDI. To unify the localization radii between all test cases, I plan to use hloc = 401 km and vloc = 1.095 log(p) for the JEDI experiments (i.e., cutoff radii). For GSI, which uses e-folding radii, these will be equivalent to hloc = 110 km and vloc = 0.30 log(p) following #67.

One problem here is that FV3-JEDI currently can only use sigma coordinates for its vertical localization (see discussion in #53). So for FV3-JEDI I just tuned the sigma radius to closely match the increment shape for a single obs test between between GSI. It seems that vloc = 0.75 sigma is very close to vloc = 0.30 log(p) in GSI.

Here is a comparison of the vloc radii chosen for each experiment. Each increment profile goes to zero at around the same level.

Edit: Here are the questions used for correlation length in GSI Var analysis (note that EnKF uses cutoff radii):

!     s_ens_h - horizontal localization correlation length of Gaussian exp(-0.5*(r/L)**2)
!               (units of km), default = 2828.0
!     s_ens_v - vertical localization correlation length of Gaussian exp(-0.5*(r/L)**2)

@TingLei-NOAA I am going off the discussion about e-folding and cutoff radius we had in #67. GSI uses these functions for localization:

!     s_ens_h - horizontal localization correlation length of Gaussian exp(-0.5*(r/L)**2)
!               (units of km), default = 2828.0
!     s_ens_v - vertical localization correlation length of Gaussian exp(-0.5*(r/L)**2)

Does that not mean the localization radius is set as e-folding radius?

E-folding scale is 2*L in the above equation. But, I got your point. Your quoted definitions of GSI rf scales could be added to your original message.

@TingLei-NOAA Great, thanks! I will update my original post.

Yes, In GSI we have two parts, the GSI part and the EnKF part. The EnKF part uses the cutoff radius as @TingLei-NOAA mentioned.