Added yaml file that include bias correction for atms_npp.

[xyzemc]

Addressing issue#52, this pull request include a new yaml file includes the bias correction part on top of rrfs_mpasjedi_2022052619_atms_npp_qc.yaml. This new yaml file named as 'rrfs_mpasjedi_2022052619_atms_npp_qc_bc.yaml'
Also, three fixed files have been added into the test data Data/obs directory:

• atms_npp.satbias.nc4
• atms_npp.satbias_cov.nc4
• atms_npp.tlapse.txt

There will be a new output directory Data/bc, which contains two output bias files

• out_atms_npp.satbias.nc
• out_atms_npp.satbias_cov.nc

The ctest has been successfully passed, the test result as bellow:

[Xiaoyan.Zhang@hfe03 rrfs-test]$ ctest
Test project /scratch2/NCEPDEV/fv3-cam/Xiaoyan.Zhang/noscrub/JEDI/RDASApp_atms/build/rrfs-test
Start 1: rrfs_mpasjedi_2022052619_Ens3Dvar
1/4 Test #1: rrfs_mpasjedi_2022052619_Ens3Dvar ......... Passed 90.76 sec
Start 2: rrfs_mpasjedi_2022052619_letkf
2/4 Test #2: rrfs_mpasjedi_2022052619_letkf ............ Passed 51.20 sec
Start 3: rrfs_mpasjedi_2022052619_atms_npp_qc
3/4 Test #3: rrfs_mpasjedi_2022052619_atms_npp_qc ...... Passed 90.54 sec
Start 4: rrfs_mpasjedi_2022052619_atms_npp_qc_bc
4/4 Test #4: rrfs_mpasjedi_2022052619_atms_npp_qc_bc ... Passed 155.86 sec
100% tests passed, 0 tests failed out of 4
Label Time Summary:
mpi = 388.37 secproc (4 tests)
rdas-bundle = 388.37 secproc (4 tests)
script = 388.37 sec*proc (4 tests)
Total Test time (real) = 388.72 sec

[SamuelDegelia-NOAA]

Hi @xyzemc, I am not seeing the yaml file in this PR yet.

[xyzemc]

Hi @xyzemc, I am not seeing the yaml file or observations in this PR yet.

Thanks for checking. It should be there now.

[SamuelDegelia-NOAA]

I copied the new atms fixed files for bias correction onto my staged data on Hera (/scratch1/NCEPDEV/da/Samuel.Degelia/RRFS/rrfs-test-data/rrfs-data_mpasjedi_2022052619/Data/obs). I will put the data on Orion whenever that maintenance ends. @guoqing-noaa will need to copy the new files into the corresponding directories on Jet. Hopefully we will have a role account soon that will make this process a little easier.

[ShunLiu-NOAA]

@guoqing-noaa Do you get a chance to update the file on JET?

[guoqing-noaa]

@guoqing-noaa Do you get a chance to update the file on JET?

/scratch1/NCEPDEV/da/Samuel.Degelia/RRFS/rrfs-test-data/rrfs-data_mpasjedi_2022052619/Data/obs has been copied to Jet. Thanks!

[ShunLiu-NOAA]

@xyzemc Could you please test your modification with the new MPAS test cases?

[xyzemc]

@xyzemc Could you please test your modification with the new MPAS test cases?

will do

[xyzemc]

@xyzemc Could you please test your modification with the new MPAS test cases?

I tried test it with new MPAS 12-km test case on 2024052700, but there is no atms_npp data cover the domain at this time. So the test can't be finished.

[guoqing-noaa]

@xyzemc will the Atlantic bigger domain have some atms_npp data?

[xyzemc]

@xyzemc will the Atlantic bigger domain have some atms_npp data?

For the case on 2024052700, the RAP ATMS bufr data only have one swath that is over the pacific ocean. I can use global gdas bufr data to test your case. I will update the result here once I get it done.
The global atms_npp data at 2024052700 coverage is here:

But the RAP atms_npp data at 2024052700 coverage is much less:

[xyzemc]

Update for testing ATMS_NPP data in new 12-km domain:
ATMS_NPP data source: 2024052700.gdas.t00z.atms.tm00.bufr_d
Converted to IODA data : 2024052700.gdas.t00z.atms.npp.tm00.nc
The radius used to do LAMDomainCheck is 2500km

   **Obs passed QC**

     **HoFx**

        **O-B with QC**

         **Analysis Increment**

[xyzemc]

Ctest result:

ctest -R rrfs_mpasjedi_2024052700_atms_npp_qc_bc
Test project /scratch2/NCEPDEV/fv3-cam/Xiaoyan.Zhang/noscrub/JEDI/RDASApp_08072024/build/rrfs-test
Start 6: rrfs_mpasjedi_2024052700_atms_npp_qc_bc
1/1 Test #6: rrfs_mpasjedi_2024052700_atms_npp_qc_bc ... Passed 313.06 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
mpi = 313.06 secproc (1 test)
rdas-bundle = 313.06 secproc (1 test)
script = 313.06 sec*proc (1 test)

Total Test time (real) = 314.06 sec

[SamuelDegelia-NOAA]

As discussed today @xyzemc, could you update this PR to just include the new rrfs-test/validated_yamls/templates/obtype_config/atms_npp_qc_bc.yaml? We do not plan to add a ctest specifically for each observation type, so the rest of the changes are not needed at the moment. Thank you!

[SamuelDegelia-NOAA]

Hi @xyzemc thank you for making the updates! I am trying to run this for the MPAS test case by inserting the obs space section from atms_npp_qc_bc.yaml into rrfs_mpasjedi_2024052700_Ens3Dvar.yaml. But I get the following error:

 20: Exception: lapse_rate does not exist in ufo::PredictorFactory
 20: Exception: oops::Variational<MPAS, UFO and IODA observations> terminating...
 20: Exception: level: 0

Also, I have a couple of suggested changes. Could you do the following:

1. Add the following to the final line of RDASApp/rrfs-test/scripts/link_mpasjedi_expr.sh so that we can link the CRTM coefficients into the new expr/ directory:

   ln -snf ${RDASApp}/fix/crtm/2.4.0 crtm_2.4.0

2. The rrfs_mpasjedi_2022052619_atms_npp_qc_bc.yaml yaml added in this PR is for the old MPAS-JEDI case that has been deprecated in RDASApp. Could you either (1) update this yaml for the new test case, or (2) remove this file from the PR since the addition to validated_yamls covers the same material.