How do you generate ESMFMesh files for WRF-CTSM?

[gunnartl]

Following the documentation on using CTSM with WRF, in section 3.4.1.7 Set ctsm runtime options, we are told to modify lilac_in in the following way:
atm_mesh_filename = '/glade/scratch/negins/wrf_ctsm_files/wrf2ctsm_land_conus_ESMFMesh_c20201110.nc'
lnd_mesh_filename = '/glade/scratch/negins/wrf_ctsm_files/wrf2ctsm_land_conus_ESMFMesh_c20201110.nc'

Could you please let us know how these files are generated?

[negin513]

@gunnartl Would you like access to these files to run the CONUS domain or are you interested in creating the mesh file for another WRF domain?
I've put the mesh file (netcdf) for the CONUS domain in dropbox so you can download it:
https://www.dropbox.com/s/hpwmk96vhjla6a3/wrf2ctsm_land_conus_ESMFMesh_c20201110.nc?dl=0

[gunnartl]

I am interested in creating the mesh file for Scandinavia. But i can use these files for some test runs. Thank you!

[mvertens]

@gunartl - if you can generate a file in SCRIP format - then there is an ESMF offline tool that will convert it to a mesh.
http://earthsystemmodeling.org/docs/nightly/develop/ESMF_refdoc/node3.html#SECTION03040000000000000000

A simple script to help you do this is below:

#!/bin/bash
#PBS -A P93300606
#PBS -N mesh
#PBS -j oe
#PBS -q share
#PBS -l walltime=03:00:00
#PBS -l select=1:mpiprocs=36
#PBS -o log.out

export MPI_USE_ARRAY=false

# load modules
module purge
module load ncarenv/1.3
module load intel/19.0.2
module load mpt/2.19
module load netcdf-mpi/4.7.1
module load pnetcdf/1.11.0
module load ncarcompilers/0.5.0
module use /glade/work/turuncu/PROGS/modulefiles/esmfpkgs/intel/19.0.2
module load esmf-8.0.0-ncdfio-mpt-O
module load nco

file_i=<full pathname of input SCRIP file>
file_o=<full pathname of output mesh file>
mpiexec_mpt `hostname` -np 8 ESMF_Scrip2Unstruct $file_i $file_o  0 ESMF

[irismuz]

Thank you @mvertens. Following your suggestion, we were able to generate the required mesh files.

In the WRF-CTSM running phase, we receive an error that indicates the inconsistency in the land-sea mask (two CPUs illustrate that this issue refers to the two tiles). Kindly see the error message below.

However, we have found no mismatch in the land-sea mask between geo_em.d01.nc and surfdata_1x1_hist_16pfts_Irrig_CMIP6_simyr2000_c210611.nc.
@negin513 could you please provide us with your input?

[ekluzek]

OK, the problem here is that CTSM needed to know how to handle ocean points. The solution she used that you see below is that she made the ocean points run as lake in CTSM.

[ekluzek]

@irismuz do you still have the landfrac mismatch you talk about above? I moved your issue to a discussion as I think that's a better forum for this kind of question.

[irismuz]

Thank you @ekluzek. The issue has been resolved by modifying ~/WRF-CTSM/CTSM/tools/contrib/create_scrip_file.ncl following the recommendation by @negin513. The following should be added to lines 23-26 of the script so that all the lake points are considered as land and that the CTSM lake model is used instead of WRF for lake points.

lake_depth = wrf_file->LAKE_DEPTH(0,:,:) 
lu_index = wrf_file->LU_INDEX(0,:,:) 
lakemask = where(lu_index.eq.21, 1,0) 
landmask = where (lakemask.eq.1,1,landmask)

[Aoe-sdh]

Thank you @ekluzek. The issue has been resolved by modifying ~/WRF-CTSM/CTSM/tools/contrib/create_scrip_file.ncl following the recommendation by @negin513. The following should be added to lines 23-26 of the script so that all the lake points are considered as land and that the CTSM lake model is used instead of WRF for lake points.

lake_depth = wrf_file->LAKE_DEPTH(0,:,:) 
lu_index = wrf_file->LU_INDEX(0,:,:) 
lakemask = where(lu_index.eq.21, 1,0) 
landmask = where (lakemask.eq.1,1,landmask)

Hi, I tried this solution but the error still exists. Are there any more moves I need to perform? Thanks.

[Aoe-sdh]

I reproduced the domain files. The error was gone. I guess all files related to geo_em.d01.nc should be rebuilt. Like wrf2clm_land.nc, domain.lnd.wrf2clm_lnd_noneg_wrf2clm_ocn_noneg.240601.nc

[negin513]

In #1892, a new python script is introduced that creates mesh files from any netcdf file with valid 1D or 2D coordinates and visualizes the mesh file../mesh_maker.py should now be used for creating mesh files from WRF domains.

Here are detailed instructions on how to use mesh_maker.py for creating a mesh file from your WRF files: https://gist.github.com/negin513/dde385a28145928786e2bc498e2b5137#creating-mesh-files-for-a-wrf-domain

For example:

./mesh_maker.py --input /glade/scratch/negins/example_files/wrfinput_d01  --output test_mesh_wrf.nc --lat XLAT --lon XLONG 

[Aoe-sdh]

Hi, I used this tool to generate mesh for WRF_CTSM. I got this error: esmf-8.4.2/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:429 ESMCI_mesh_create_from_ESMFMesh_ The NetCDF Library returned an error - Unable to open existing file: /GFPS8p/ycfan8/WRF/WRF_CTSM_Nor/CTSM_Nor/inputdata/China_SingleD_FNL_9km, (PIO/PNetCDF error = NetCDF: Attempt to use feature that was not turned on when netCDF was built.)
I checked the code which says: // if the file was created with netcdf4, it cannot be opened with pnetcdf
if (piorc == PIO_EINVAL){
pio_type = PIO_IOTYPE_NETCDF;
piorc = PIOc_openfile(pioSystemDesc, &pioFileDesc, &pio_type, filename, mode);
}
if (!CHECKPIOERROR(piorc, std::string("Unable to open existing file: ") + filename,
ESMF_RC_FILE_OPEN, localrc)) throw localrc;
Does that mean I need to generate pnetcdf files?

[Aoe-sdh]

I used esmf-8.6.1 and this was solved