IODA files not working with MET 11 #2134

davis-wesley-12345 · 2023-04-21T17:22:44Z

davis-wesley-12345
Apr 21, 2023

files are at DAVIS_DATA

MET version 11.0.1
filename = raob_all_v1_20201216T0000Z.nc4

+ singularity exec --bind /p/cwfs/esatterf/MET_share /p/cwfs/esatterf/MET_share/MET_singularity/met_v11.0.1.simg ioda2nc /p/work1/daviswj/OBS/IODA_FILES/V1/raob_all_v1_20201216T0000Z.nc4 /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2020121600/raob_all_v1_20201216T0000Z.nc -config IODA2NCConfig_hdr_typ.cfg -v 1 -log log.ioda2nc
DEBUG 1: Start ioda2nc by daviswj(927863) at 2023-04-21 16:37:21Z  cmd: /usr/local/bin/ioda2nc /p/work1/daviswj/OBS/IODA_FILES/V1/raob_all_v1_20201216T0000Z.nc4 /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2020121600/raob_all_v1_20201216T0000Z.nc -config IODA2NCConfig_hdr_typ.cfg -v 1 -log log.ioda2nc
DEBUG 1: Default Config File: /p/cwfs/esatterf/MET_share/data/met-11.0.1_share/config/IODA2NCConfig_default
DEBUG 1: User Config File: IODA2NCConfig_hdr_typ.cfg
DEBUG 1: Creating NetCDF File:          /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2020121600/raob_all_v1_20201216T0000Z.nc
DEBUG 1: Processing IODA File:  /p/work1/daviswj/OBS/IODA_FILES/V1/raob_all_v1_20201216T0000Z.nc4
ERROR  :
ERROR  : process_ioda_file() -> Fail to get datetime variable
ERROR  :

MET version 10.1.2
filename = raob_all_v1_20201216T0000Z.nc4

WORKS!

MET version 10.1.2
filename = raob_20210806T1200Z.nc4

+ singularity exec /app/projects/NEPTUNE/software/singularity/met.simg ioda2nc /p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4 /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2021080612/raob_v2_20210806T1200Z.nc -config IODA2NCConfig_hdr_typ.cfg -v 1 -log log.ioda2nc
DEBUG 1: Default Config File: /app/projects/NEPTUNE/database/MET/met-10.1.2_share/config/IODA2NCConfig_default
DEBUG 1: User Config File: IODA2NCConfig_hdr_typ.cfg
DEBUG 1: Creating NetCDF File:          /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2021080612/raob_v2_20210806T1200Z.nc
DEBUG 1: Processing IODA File:  /p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4
ERROR  :
ERROR  : process_ioda_file() -> can't open input NetCDF file "/p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4" for reading.
ERROR  :

MET version 11.0.1
filename = raob_20210806T1200Z.nc4

DEBUG 1: Start ioda2nc by daviswj(927863) at 2023-04-21 17:15:12Z  cmd: /usr/local/bin/ioda2nc /p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4 /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2021080612/raob_v2_20210806T1200Z.nc -config IODA2NCConfig_hdr_typ.cfg -v 1 -log log.ioda2nc
DEBUG 1: Default Config File: /p/cwfs/esatterf/MET_share/data/met-11.0.1_share/config/IODA2NCConfig_default
DEBUG 1: User Config File: IODA2NCConfig_hdr_typ.cfg
DEBUG 1: Creating NetCDF File:          /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2021080612/raob_v2_20210806T1200Z.nc
DEBUG 1: Processing IODA File:  /p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4
ERROR  :
ERROR  : check_core_data() -> -> core variable  "datetime" is missing.
ERROR  :
ERROR  :
ERROR  : process_ioda_file() -> Please check the IODA file (required dimensions or meta variables are missing).
ERROR  :

ncdump -h raob_20210806T1200Z.nc4

        int64 dateTime(nlocs) ;
                dateTime:_FillValue = -9223372036854775806LL ;
                string dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
                string dateTime:coordinates = "longitude latitude" ;

ncdump -h raob_all_v1_20201215T0600Z.nc4

char datetime@MetaData(nlocs, ndatetime) ;

Answered by davis-wesley-12345

May 2, 2023

I upped the debug verbosity and got this:

DEBUG 2: Processing air_temperature/250.0 versus air_temperature/P25000, for observation type RAOB, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 267050
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 213640
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 5957
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 47064
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 389
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad …

View full answer

j-opatz · 2023-04-24T22:13:53Z

j-opatz
Apr 24, 2023
Collaborator

Hi Davis,

And thank you for your question. I just ran both of the files you provided us (raob_all_v1_20201216T0000Z.nc4 and raob_20210806T1200Z.nc4) through ioda2nc and was unable to duplicate the behavior you're seeing.

The first file I tested, raob_all_v1_20201216T0000Z.nc4, had no issues with the default ioda2nc configuration file for the 4 MET releases I tested (10.0.0, 10.1.1, 11.0.0, and 11.0.1). Here's the readout from one of the version 11 runs:

DEBUG 1: Default Config File: /usr/local/met-11.0.1/share/met/config/IODA2NCConfig_default
DEBUG 1: Creating NetCDF File:		testout_11_0_1.nc
DEBUG 1: Processing IODA File:	raob_all_v1_20201215T1200Z.nc4
DEBUG 2: Processing 88290 IODA messages...
DEBUG 2:  IODA Time Center:		20201215_111600
DEBUG 2: Searching Time Window:		20201215_094600 to 20201215_124600
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 

DEBUG 2: Total records processed		= 88290
DEBUG 2: Rejected based on message type		= 0
DEBUG 2: Rejected based on station id		= 0
DEBUG 2: Rejected based on valid time		= 0
DEBUG 2: Rejected based on masking grid		= 0
DEBUG 2: Rejected based on masking polygon	= 0
DEBUG 2: Rejected based on elevation		= 0
DEBUG 2: Rejected based on zero observations	= 0
DEBUG 2: Total Records retained			= 88290
DEBUG 2: Total observations retained or derived	= 706320
DEBUG 1: Obs time between 20201215_110427 and 20201215_123927

When I ran ioda2nc with the second file, raob_20210806T1200Z.nc4, for the same 4 MET releases I did run into errors, but only in the MET version 10 releases. This is exactly opposite of what you experienced. Those errors looked like the following:

DEBUG 1: Default Config File: /d1/projects/MET/MET_releases/met-10.0.2/share/met/config/IODA2NCConfig_default
DEBUG 1: Creating NetCDF File:		testout_10_0_2.nc
DEBUG 1: Processing IODA File:	raob_20210806T1200Z.nc4
ERROR  : 
ERROR  : check_core_data() -> -> core dimension "nvars" is missing.
ERROR  : 
ERROR  : 
ERROR  : check_core_data() -> -> core variable  "datetime" is missing.
ERROR  : 
ERROR  : 
ERROR  : check_core_data() -> -> core variable  "latitude" is missing.
ERROR  : 
ERROR  : 
ERROR  : check_core_data() -> -> core variable  "longitude" is missing.
ERROR  : 
ERROR  : 
ERROR  : process_ioda_file() -> Please check the IODA file (required dimensions or meta variables are missing).

Because the MET 10 versions are an older version, it's not surprising to see that they are not able to process files in the same way as newer versions of MET do. However, the flipped behavior I'm seeing to what you're describing is something we can look more into.

Would you be able to try a few things and report back how they go? The first is to try to rerun the file(s) using only the default configuration file for each release. That can be accomplished by simply omitting the configuration file from the ioda2nc prompt, i.e. ioda2nc path/to/file_in.nc path/to/file_out.nc.
The second would be to increase verbosity of the run (4 should suffice) and provide the output in your reply. It looks like you were running at a verbosity of 1 before. You can also attach the log file from the run result at the higher verbosity and that might help us diagnose the issue.

1 reply

davis-wesley-12345 May 1, 2023
Author

MET v11.0.1
Default Config and no other

DEBUG 1: Creating NetCDF File: /p/work1/daviswj/uv/scripts/NEPTUNE/tmp_dir/verif/2021080612/raob_v2_20210806T1200Z.nc
DEBUG 1: Processing IODA File: /p/work1/daviswj/OBS/IODA_FILES/V2/raob_20210806T1200Z.nc4
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "seconds since 1970-01-01T00:00:00Z"
DEBUG 4: as a reference time of 19700101_000000 and 1 second(s) per time step.
DEBUG 4: get_meta_data_float() -> Metadata for height does not exist!
DEBUG 4: get_meta_data_float() -> Metadata for elevation does not exist!
DEBUG 1: process_ioda_file() -> The metadata variable for message type does not exist!
DEBUG 4: get_obs_data_float() -> "mixing_ratio" does not exist!
DEBUG 3: PERF: process_ioda_file() 2.64 seconds for preparing
DEBUG 2: Processing 102676 IODA messages...
DEBUG 2: IODA Time Center: 20210806_114400
DEBUG 2: Searching Time Window: 20210806_101400 to 20210806_131400
DEBUG 2: Total records processed = 102676
DEBUG 2: Rejected based on message type = 0
DEBUG 2: Rejected based on station id = 0
DEBUG 2: Rejected based on valid time = 2505
DEBUG 2: Rejected based on masking grid = 0
DEBUG 2: Rejected based on masking polygon = 0
DEBUG 2: Rejected based on elevation = 0
DEBUG 2: Rejected based on zero observations = 0
DEBUG 2: Total Records retained = 100171
DEBUG 2: Total observations retained or derived = 500855

ncdump raob_20210806T1200Z.nc4 > raob_contents_full.txt
less raob_contents_full.txt
search for dateTime

dateTime = 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250240, 1628250240, 1628250240, 1628250240, 1628250240, 1628250240,
1628250819, 1628250731, 1628250641, 1628250570, 1628250469, 1628250373,
1628250241, 1628250248, 1628250084, 1628250005, 1628249742, 1628249762,
1628249599, 1628249613, 1628249642, 1628249480, 1628249442, 1628249380,
1628249195, 1628249205, 1628249150, 1628249037, 1628249076, 1628248910,
1628248652, 1628248541, 1628248547, 1628248418, 1628248271, 1628248093,
1628248104, 1628248030, 1628247921, 1628247891, 1628247755, 1628247637,
1628247620, 1628247627, 1628247600, 1628247605, 1628247614, 1628247720,
1628247720, 1628247720, 1628247734, 1628247765, 1628247798, 1628247818,
1628247831, 1628247861, 1628247886, 1628247915, 1628247942, 1628247954,
1628247974, 1628248001, 1628248028, 1628248057, 1628248085, 1628248114,

From the default config in met-11.0.1_share/config/IODA2NCConfig_default

obs_window = {
   beg = -5400;
   end =  5400;
}

Is the obs_window too tight maybe?

hsoh-u · 2023-04-25T14:58:35Z

hsoh-u
Apr 25, 2023
Collaborator

The ioda file with the NetCDF group (IODA version 2) is supported from MET 11.
In NetCDF, "dateTime" variable in the root group (when the group is not assigned) and "dateTime" in the specific group are different variables.

netcdf raob_20210806T1200Z {
dimensions:
        nlocs = UNLIMITED ; // (102676 currently)
        nrecs = 55137 ;
variables:
        int nlocs(nlocs) ;
                nlocs:suggested_chunk_dim = 102676LL ;
        int nrecs(nrecs) ;
                nrecs:suggested_chunk_dim = 55137LL ;

// global attributes:
...

group: MetaData {
  variables:
        float air_pressure(nlocs) ;
                air_pressure:_FillValue = -3.368795e+38f ;
                string air_pressure:coordinates = "longitude latitude" ;
        int64 dateTime(nlocs) ;
                dateTime:_FillValue = -9223372036854775806LL ;
                string dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
                string dateTime:coordinates = "longitude latitude" ;
        float latitude(nlocs) ;
                latitude:_FillValue = -3.368795e+38f ;
                string latitude:coordinates = "longitude latitude" ;
                string latitude:units = "degrees_north" ;
        float longitude(nlocs) ;
                longitude:_FillValue = -3.368795e+38f ;
                string longitude:coordinates = "longitude latitude" ;
                string longitude:units = "degrees_east" ;
        ...
}
group: ObsError {
  variables:
        float air_temperature(nlocs) ;
        ...
}
group: ObsValue {
  variables:
        float air_temperature(nlocs) ;
        ...
}
group: PreQC {
  variables:
        int air_temperature(nlocs) ;
        ...
}

0 replies

davis-wesley-12345 · 2023-05-02T23:29:57Z

davis-wesley-12345
May 2, 2023
Author

I upped the debug verbosity and got this:

DEBUG 2: Processing air_temperature/250.0 versus air_temperature/P25000, for observation type RAOB, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 267050
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 213640
DEBUG 2: Rejected: valid time = 0
DEBUG 2: Rejected: bad obs value = 5957
DEBUG 2: Rejected: off the grid = 0
DEBUG 2: Rejected: topography = 0
DEBUG 2: Rejected: level mismatch = 47064
DEBUG 2: Rejected: quality marker = 0
DEBUG 2: Rejected: message type = 389
DEBUG 2: Rejected: masking region = 0
DEBUG 2: Rejected: bad fcst value = 0
DEBUG 2: Rejected: bad climo mean = 0
DEBUG 2: Rejected: bad climo stdev = 0
DEBUG 2: Rejected: mpr filter = 0
DEBUG 2: Rejected: duplicates = 0

DEBUG 2: Rejected: message type = 389

This stuck out so I checked what was in the post ioda2nc file under message type and got this:

ncdump tmp_dir/verif/2021081200/raob_v2_20210812T0000Z.nc > tmp_dir/verif/2021081200/post_ioda.nc

hdr_typ_table =
"NA" ;

So I looked in my point_stat config file and added NA to my message_type_group_map

message_type_group_map = [
   { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";               },
   { key = "ANYAIR";  val = "AIRCAR,AIRCFT";                      },
   { key = "ANYSFC";  val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
   { key = "ONLYSF";  val = "ADPSFC,SFCSHP";                      },
   { key = "LANDSF";  val = "ADPSFC,MSONET";                      },
   { key = "WATERSF"; val = "SFCSHP";                             },
   { key = "RAOB";    val = "n_rbraob,n_rlraob,n_pbpibal,n_plpibalB,n_plpibalD,n_plpibalA,n_plpibalC,NA";}
];

And success!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IODA files not working with MET 11 #2134

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

IODA files not working with MET 11 #2134

davis-wesley-12345 Apr 21, 2023

WORKS!

Replies: 3 comments · 1 reply

j-opatz Apr 24, 2023 Collaborator

davis-wesley-12345 May 1, 2023 Author

hsoh-u Apr 25, 2023 Collaborator

davis-wesley-12345 May 2, 2023 Author

davis-wesley-12345
Apr 21, 2023

Replies: 3 comments 1 reply

j-opatz
Apr 24, 2023
Collaborator

davis-wesley-12345 May 1, 2023
Author

hsoh-u
Apr 25, 2023
Collaborator

davis-wesley-12345
May 2, 2023
Author