A basic sample Yaml file for the new integrated operator for surface observations with consistent adjustments

[HuiLiu-NOAA]

A new UFO operator "SfcCorrected" is proposed to integrate surface temperature, humidity, and wind observations (and pressure later) with various adjustments. The main motivation is to ensure consistent empirical adjustments to simulations (HofX) and optimal assimilation of the observations. It would also enable simpler configurations of the observations in a yaml file.
First, a few terrain height adjustments for T2m are included. The options of "WRFDA" and "UKMO" are consistent with the air temperature assumptions between model surface and observation station height used in the existing surface pressure operator "SfcPCorrected". The "GSL" option is intended for the experimental Rapid Refresh Forecast System (RRFS). The existing pressure operator could be blended into this new operator later.
For specific humidity and wind, empirical height adjustments could be evaluated and added later, like the UKMO options in the obsfunctions/.
Wind is simulated from model surface level and scaled by a ratio of model diagnosed 10m wind over model surface wind, which is calculated online in JEDI.

This basic yaml could serve as a template to add options for specific obs types and purpose.

Dependent:
https://github.com/JCSDA-internal/ufo/pull/3440

[guoqing-noaa]

@HuiLiu-NOAA Thanks for the advances on this.
Sijie (@spanNOAA) is on leave until 9/23. Do you want to wait for him to review?

We can also review and merge this now and update it later if needed.

[HuiLiu-NOAA]

@HuiLiu-NOAA Thanks for the advances on this. Sijie (@spanNOAA) is on leave until 9/23. Do you want to wait for him to review?

We can also review and merge this now and update it later if needed.

No problem for me.

[SamuelDegelia-NOAA]

Thanks @HuiLiu-NOAA for adding the new operator! My only question until @spanNOAA reviews is do we want to keep this as a separate yaml, or integrate it into the other mesonet yaml files msonet_airTemperature_obtype188.yaml, msonet_specificHumidity_obtype188.yaml, etc.

[HuiLiu-NOAA]

@SamuelDegelia-NOAA : I guess it might be better to update the mesonet yaml files to use the new operator, since the vertical interpolation operator does not have the adjustments in place.

[delippi]

@HuiLiu-NOAA, Thanks for working on this new surface observation operator!

@SamuelDegelia-NOAA and @HuiLiu-NOAA While updating the mesonet YAMLs to use this new operator will be necessary eventually, I believe there are still critical steps we need to complete first. Before moving on to new observation operators and jumping into MPAS-JEDI, we need to ensure that our FV3-JEDI validation is thoroughly aligned with GSI. Specifically, we should focus on replicating GSI as closely as possible without relying on GSI inputs (e.g., GSI-IODA and GSI-geovals) and confirming that all other observation filters and settings are correct. Only after we are confident that FV3-JEDI is validated properly should we introduce new components. I think this approach will help avoid complications later. The "validated" mesonet YAMLs without the new observation operator should be good enough for now to use for cycling MPAS experiments. I think it is fine to keep as separate YAML, for now. Just my two-cents.

[TingLei-NOAA]

For me, I would like keeping all those (old and new) yaml files available to rdasapp users/developers for following comparison/test/evaluation and, not the least, learning.

[HuiLiu-NOAA]

I agree with @TingLei-NOAA .

[SamuelDegelia-NOAA]

Since we talked about splitting the validated_yaml directory up into an MPAS and FV3-JEDI directory, how about using the new observation operator for the MPAS-JEDI section? @HuiLiu-NOAA showed large biases in surface temp without the operator, so I am of the opinion that we include the operator with any cycling experiments.

[HuiLiu-NOAA]

@SamuelDegelia-NOAA : This sounds a good idea to me. Maybe to separate these yaml files into categories of "testing" etc according to their purpose?

[delippi]

Since we talked about splitting the validated_yaml directory up into an MPAS and FV3-JEDI directory, how about using the new observation operator for the MPAS-JEDI section? @HuiLiu-NOAA showed large biases in surface temp without the operator, so I am of the opinion that we include the operator with any cycling experiments.

@SamuelDegelia-NOAA I'm fine with that. But also note that those "validated" mesonet yamls that will be in the MPAS directory won't yet have things like time filtering or even ob error inflation for example. There's probably a number of things that haven't been considered yet either because I haven't done the tests without GSI-IODA/geovals. I'm currently trying to "just get them to work" for MPAS-JEDI--then go back and do more rigorous validation. My understanding of the cycling experiments, for the moment, is to show that it is functioning and not so much the science.

[SamuelDegelia-NOAA]

That is true @delippi. Either way though we at least need to start getting some yamls ready for cycling (even if they are missing some things) instead of either the single obs type version that we have been using or those specific to fv3-jedi validation in validated_yamls. But we can add this yaml as is in the PR (once reviewed by @spanNOAA), and afterwards we can start working on creating a new directory of mpasjedi-specific yamls ready for cycling. I can plan to do that when building the updated ctests that will use more observation types.

[delippi]

First of all, thanks for adding this new operator. This is definitely important work.

This yaml file looks fairly generic for any surface observation type. What observation errors are being used? Are they the values in the BUFR/IODA file, or are they what GSI would use? Those might not be the same! I think it would be best to define those observation errors directly in the yaml, so we can easily see what’s being used and adjust them later if needed. Also, how about observation error inflation? It might be useful to separate airTemperature, specificHumidity, winds, and surface pressure (sfcp) into their own yaml files (e.g., adpsfc_airTemperature_187.yaml, etc.) to prevent the configuration from getting too long. I'm not sure how JCB works, but separating them like this could make transitioning easier and help with debugging and validation.

I had intended the "validated_yamls" directory to house yamls that have already been verified to achieve one-to-one H(x) and observation error matching between FV3-JEDI and GSI. This means performing a sanity check to ensure the configuration matches that baseline. Otherwise, it’s hard to be sure there isn't a misconfiguration. Once that validation is done, I think it’s fine to incorporate this new operator into the yaml files. Has that validation been completed for GSI vs FV3-JEDI?

[HuiLiu-NOAA]

Thanks @delippi for the comments.
I agree that this is a basic/generic yaml file. The obs error is obtained from the obs input file as default. You may add more options to it for specific obs types and purposes.

[delippi]

Thanks @delippi for the comments. I agree that this is a basic/generic yaml file. The obs error is obtained from the obs input file as default. Users have freedom to add more options to it for specific obs types and purposes.

Okay, I understand that the observation error is obtained from the input file by default. While users have the flexibility to add more options for specific obs types and purposes, I think one of the key reasons for creating these yaml files is to support the RDASApp user and its specific purpose.

With that in mind, I believe it’s important to explicitly define the observation errors in the yaml so that we don’t have to track them down later (make sure to use the correct errtable; I used the table from RRFS_A which is different from the test case for some obs). I also believe it would be best to split them into several yamls for the specific obtypes that you have tried as well as adding the oberror inflation functions.

Let's see what @ShunLiu-NOAA thinks about splitting them into several yamls if he has any recommendation?

[HuiLiu-NOAA]

thanks @delippi for the suggestion.
I agree that it is indeed necessary to have specific yaml files for each of the specific surface obs type you have tested and validated based on this basic yaml (as a template, which is the purpose of this PR). I have done little in this aspect yet.

[HuiLiu-NOAA]

The title of this PR is changed to "A basic sample Yaml file for the new integrated operator for surface observations with consistent adjustments", which would be more clear for its purpose.

[SamuelDegelia-NOAA]

@delippi Since we are not ready to merge this into any of the validated yamls, would you prefer to just put this in rrfs-test/testinput for now?

[delippi]

@HuiLiu-NOAA @SamuelDegelia-NOAA, I like the idea to put this somewhere else since this is just a basic template. It is really good to have this template available for future reference. I'm just not sure where the best place would be. rrfs-test/testinput is probably fine.

[delippi]

@HuiLiu-NOAA, I'm just curious if have you looked at the resulting increments from tests using this yaml?

I also didn't know if you knew this PR was still in draft mode? And also, I think we just need to move this generic yaml to somewhere that isn't the validated_yamls/templates/obtype_config location. The previously mentioned location rrfs-test/testinput is fine or a new directory under validated_yamls/templates is also probably fine to keep it somewhat close to the validated yamls.

[HuiLiu-NOAA]

Thanks @delippi.

1. yes, I looked at the resulting increments from tests using this yaml in the OLD MPAS test case and it looked reasonable. But, I would like to test the yaml with the new MPAS test case when the corrected ioda files are available. This is one reason for the "draft" status of the PR.
2. Another reason for the "draft" status is that it may be better to wait for Sijie's test of this new operator when he is back. I am also open to where the yaml may sit. rrfs-test/testinput is fine to me. Other suggestions are welcome.

[delippi]

@HuiLiu-NOAA, by OLD MPAS case you mean the case with 5 members, right? It is good to know that you are waiting on those updated ioda for this. I will try to get working on that asap. In the mean time, I think you could try just assimilating airTemperature as those ObErrors defined within the ioda file should work, yes? It would probably be really good to test each obtype individually anyway. You can also try overriding using the ioda-defined ObErrors by setting ObErrors yourself too. That way you're not stuck waiting on me to get those new ioda files. Was there something else aside from the ObErrors that is giving you problems?

[HuiLiu-NOAA]

According to @guoqing-noaa, @spanNOAA will look into this issue?

[HuiLiu-NOAA]

see #169

[guoqing-noaa]

@HuiLiu-NOAA If it is not urgent, we can let @spanNOAA take a look. He will be back next Monday.

[HuiLiu-NOAA]

@guoqing Ge - NOAA Affiliate ***@***.***> . Thanks!. This is perfect for me.

…

On Thu, Sep 19, 2024 at 1:49 PM Guoqing Ge ***@***.***> wrote: @HuiLiu-NOAA <https://github.com/HuiLiu-NOAA> If it is not urgent, we can let @spanNOAA <https://github.com/spanNOAA> take a look. He will be back next Monday. — Reply to this email directly, view it on GitHub <#152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOF252B425I6YYAAL4L4343ZXMTEDAVCNFSM6AAAAABNZBLQQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRSGA2TCNZZG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[guoqing-noaa]

@spanNOAA Could you review this PR? Thanks!

[spanNOAA]

Thanks @HuiLiu-NOAA for your efforts on this YAML file and for all the discussions surrounding it. Everything looks good to me!

[HuiLiu-NOAA]

Thanks @guoqing-noaa and @spanNOAA.

@guoqing-noaa : could you clone the latest version of the operator and share with Sijie?

[delippi]

Thanks @delippi.

1. yes, I looked at the resulting increments from tests using this yaml in the OLD MPAS test case and it looked reasonable. But, I would like to test the yaml with the new MPAS test case when the corrected ioda files are available. This is one reason for the "draft" status of the PR.
2. Another reason for the "draft" status is that it may be better to wait for Sijie's test of this new operator when he is back. I am also open to where the yaml may sit. rrfs-test/testinput is fine to me. Other suggestions are welcome.

@HuiLiu-NOAA, please go ahead and move it into rrfs-test/testinput. Have you had a chance to use the new IODA files? Maybe I haven't shared their location yet. If not, here is a temporary location for them:
/scratch2/NCEPDEV/fv3-cam/Donald.E.Lippi/RRFSv2/ioda_processing/ioda