Duplicate Checking in RDAS

[delippi]

Description

In GSI, RRFS uses l_closeobs in the setup routines to perform duplicate checks. This option retains the observation at a given lat/lon/pressure that is closest to the analysis time. Currently, we have been using Temporal Thinning filter in JEDI to perform duplicate checks; however, we have been specifying the category variable (i.e., how the observations are grouped) as MetaData/stationIdentification which will not always work. For example, if you have a profiler where there may be many observations at the same lat/lon/stationID but different pressure, only a single observation will be retained from that profile. The category variable must be a string or integer and it can't [currently] be a list of variables. I thought my proposed solution below would be easier than updating the code to be able to use more than one category variable.

Proposed fix:
We can add an ad hoc variable such as a string like "longitude_latitude_pressure" to the IODA file to be used as the category variable. This would be best suited to be placed in a python bufr2ioda converter; however, we are currently using yaml based converters. Therefore, I am proposing to add this to the offline_domain check code and we will just run that offline tool for the early stages of development.

Acceptance Criteria (Definition of Done)

The Temporal Thinning filter can use longitude latitude and pressure (not just station ID) as category variable either by adding the ad hoc variable to the offline domain check, a separate tool, python converter, or updating the Temporal Thinning code.

• Link any relevant pull requests here:
  • PR #
  • PR #

Dependencies

None

[delippi]

Are there any objections to adding the variable to the offline domain check and just using that for early cycling experiments? That is likely the easiest way forward until a more permanent solution can be made.