accessing elevation data in IODA v2

[Wesley-J-Davis]

I am having trouble accessing elevation data in IODAv2 format so as to compare air temp and winds from buoy and surface observation data against a model.

Supporting data and configurations and scripts can be found under DAVIS_DATA on the ftp.

the forecast data is found under fcst and the reader for it is read_NRL_FLATFILE_CONVERT_CtoK.py.

The observation data is surface_ship_buoy.nc4. This is converted through IODA2NC to buoys_2021060500.nc4. Reading from this file in point_stat has failed due to not being able to access elevation data.

the data in the obs folder can mostly be disregarded, I just didn't know how to delete it. Within it though you will find the data's original source, the h5 files and then the .nc4 files are the result of an NRL conversion script which turns the h5's into IODAv2. It is this IODAv2 format I am having trouble accessing data using elevation parameters.

OUTPUT FROM POINT STAT INDICATES LEVEL MISMATCH IS THE PRIMARY CULPRIT

DEBUG 2: Processing airtmp/zht_0002.0 versus air_temperature/Z0.1-2, for observation type RAOB, over region SOUTHERN_HEMISPHERE, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 207792
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 181818
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 10376
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 15598
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 0
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad fcst value = 0
DEBUG 2: Rejected: bad climo mean = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter = 0
DEBUG 2: Rejected: duplicates = 0

// Point-Stat configuration file.
obtype = "RAOB";
obs = { field = [ {
name = "eastward_wind";
level = ["Z10-.1"];
cat_thresh = [ NA ];
is_u_wind = TRUE; },
{
name = "northward_wind";
level = ["Z10-.1"];
cat_thresh = [ NA ];
is_v_wind = TRUE; },
{
name = "air_temperature";
level = ["Z2-.1"]; // this nomenclature searches elevation (Z) from .1-2. This is GRIB searching nomenclature
// which SHOULD work with output from IODA2NC since this is a certified MET format.
// it certainly does for pressure levels.
cat_thresh = [ NA ]; } ];
message_type = ["RAOB"];
message_type_group_map = [
{ key = "RAOB"; val = "n_rbraob,n_rlraob,n_pbpibal,n_plpibalB,n_plpibalD,n_plpibalA,n_plpibalC,NA";},]

From IODA2NC output (WHAT POINT STAT IS READING)
float hdr_elv(nhdr) ;
hdr_elv:long_name = "elevation" ;
hdr_elv:_FillValue = -9999.f ;
hdr_elv:units = " (msl)" ;
float obs_lvl(nobs) ;
obs_lvl:long_name = " (hPa) or accumulation (sec)" ;
obs_lvl:_FillValue = -9999.f ;
float obs_hgt(nobs) ;
obs_hgt:long_name = "meters above sea level (msl)" ;
obs_hgt:_FillValue = -9999.f ;

hdr_typ_table = "NA" ;
obs_lvl = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
obs_hgt = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
hdr_elv = 0, 0.1, 0, 0, 0, 0, 0.1, 0, 0, 0, 0.1, 0, 0, 0.1, 0, 0, 0, 0, 0,

[hsoh-u]

hdr_elv is the elevation of the station which is a metadata. hdr_elv is not used for the level matching. obs_lvl or obs_hgt which are all missing values is used to match the level.
As default, ioda2nc sets obs_lvl from the variable "air_pressure" or "pressure" at the ObsValue group. obs_hgt is filled from the variable height or height_above_mean_sea_level at the ObsValue group. The IOD2NC has a configuration to find the pressure level and height.

Step 1. Please add the proper pressure variable name into metadata_map at IODA2NCConfig_BUOYS. For example, "surface_pressure" is added as the pressure level.

metadata_map = [
   { key = "message_type"; val = "msg_type,station_ob"; },
   { key = "station_id";   val = "station_id,report_identifier"; },
   { key = "pressure";     val = "air_pressure,pressure"; },
   { key = "height";       val = "height,height_above_mean_sea_level"; },
   { key = "datetime";     val = "datetime,dateTime"; },
   { key = "elevation";    val = "elevation,station_elevation"; }
];
==>
metadata_map = [
   { key = "message_type"; val = "msg_type,station_ob"; },
   { key = "station_id";   val = "station_id,report_identifier"; },
   { key = "pressure";     val = "air_pressure,pressure,surface_pressure"; },
   { key = "height";       val = "height,height_above_mean_sea_level"; },
   { key = "datetime";     val = "datetime,dateTime"; },
   { key = "elevation";    val = "elevation,station_elevation"; },
   { key = "nlocs";        val = "Location"; }
];

Step 2. Re-run ioda2nc with -config IODA2NCConfig_BUOYS option.

[Wesley-J-Davis]

Having made those changes and reran using the same configurations for everything else, I get the same results.

DEBUG 2: Processing airtmp/zht_0002.0 versus air_temperature/Z0.1-2, for observation type RAOB, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 207792
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 181818
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 10376
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 15598
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 0
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad fcst value = 0
DEBUG 2: Rejected: bad climo mean = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter = 0
DEBUG 2: Rejected: duplicates = 0

Sounds like the real issue is that the obs_hgt and obs_lvl are unpopulated and that's something we need to handle in our conversion scripts.

[hsoh-u]

The obs_hgh and obs_lvl variables are filled from the variables in the MetaData group. The variable surface_pressure exists in the ObsValue group. That's why there is no differences with the configuration changes. The quick solution is copying surface_pressure in the ObsValue group to the MetaData group.
I modified two files and they are available under hsoh_data directory on the ftp:

• IODA2NCConfig_BUOYS: disabled filtering by station_id
• surface_ship_buoy_new.nc4: generated by following way:

1. Save the netcdf file as text form by using ncdump: ncdump obs/surface_ship_buoy.nc4 > obs/surface_ship_buoy.nc4.dump
2. Open obs/surface_ship_buoy.nc4.dump by using a text editor
3. Copy surface_pressure variable (both header and data) in the ObsValue group to the MetaData group
4. Save obs/surface_ship_buoy.nc4.dump
5. Generate the NetCDF file by using ncgen: ncgen -o obs/surface_ship_buoy_new.nc4 obs/surface_ship_buoy.nc4.dump

[Wesley-J-Davis]

So just to be clear here, if I have only pressure data and no height data, can I search for data by height?

Even if everything got processed correctly on our end and the surface pressure data that was in the ObsValue group made it to the MetaData group in the IODA file, that would still mean the obs_hgh variable would be empty.

If I searched that dataset for 2m winds using the level=['Z2']; format without elevation data, would it:

1. Still throw an error or
2. Would MET parse the obs_lvl and convert it to height_above_mean_sea_level or height?