mode - ERROR: Trouble reading observation file

[Wesley-J-Davis]

Tested the file with plot_data_plane and was able to get a good postscript output.
code for good plot_data_plane output:
(base) [davis@lorenz MODE]$ plot_data_plane cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc test.ps 'name="cloud_area_fraction"; level="(0,0,*,*)"; file_type=NETCDF_NCCF;' -v 4 DEBUG 1: Opening data file: cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF". DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1800-1-1 00:00:0.0" DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1800-1-1 00:00:0.0" as a reference time of 18000101_000000 and 3600 second(s) per time step. DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable. DEBUG 4: NcCfFile::open() -> could not extract init time from file name. DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF". DEBUG 4: NcCfFile::getData() -> setting the unset init time to the valid time of 20200511_120000. DEBUG 4: DEBUG 4: Data plane information: DEBUG 4: plane min: 0 DEBUG 4: plane max: 0.99975 DEBUG 4: valid time: 20200511_120000 DEBUG 4: lead time: 000000 DEBUG 4: init time: 20200511_120000 DEBUG 4: accum time: 000000 DEBUG 1: Creating postscript file: test.ps DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/country_data DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/usa_state_data DEBUG 4: draw_map() -> Plotting map data file: /software/depot/met-9.1/met-9.1/share/met/map/major_lakes_data

Tried using mode to read that file as both forecast and obs and got the following error:

mode cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc template.conf -outdir . -v 4 DEBUG 1: Default Config File: /software/depot/met-9.1/met-9.1/share/met/config/MODEConfig_default DEBUG 1: Match Config File: template.conf DEBUG 1: Merge Config File: template.conf DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None". ERROR : ERROR : Trouble reading observation file "cloud_area_fraction.e016.sky_cvr.0000.0.glob144x73.2020051012.daily_avg_12Z_12h.nc" ERROR :
Data related to this issue is available at : "/incoming/irap/met_help/DAVIS_data"

sample of forecast and ob configuration:

field =
 { name = "cloud_area_fraction";
   level = "(0,0,*,*)";
   file_type = NETCDF_NCCF ;

   censor_thresh      = [];
   censor_val         = [];
   conv_radius       = [100/grid_res];
   conv_thresh       = [>=75];
   vld_thresh        = 0.5;
   filter_attr_name   = [];
   filter_attr_thresh = [];
   merge_thresh      = [>=65];
   merge_flag        = BOTH;
};
}

obs=fcst;
//obs = {
//field =
// { name = "cloud_area_fraction";
//   level = "(1,0,*,*)";
//   file_type = NETCDF_NCCF ;
//
//   conv_radius       = [100/grid_res];
//   conv_thresh       = [>=75];
//   vld_thresh        = 0.5;
//   merge_thresh      = [>=65];
//   merge_flag        = BOTH;
//};
//}

Any help is greatly appreciated!

[j-opatz]

Hello, and thank you for your question.

I was able to download your files (thank you for utilizing our ftp server for that data transfer!) and confirmed the behavior you described. When you created the plot_data_plane output, does the resulting image look correct? My concern is that while the file_type option helps your file to be understood by plot_data_plane, it's not a true netCDF-CF compliant file. You can always write up a Python script to ingest your netCDF file and then feed it into MET tools, but that doesn't answer the question of why it's working for one tool, and not another.

I'll defer to @JohnHalleyGotway on this issue and the potential solution of not being able to read in this file.

[Wesley-J-Davis]

does the resulting image look correct

It actually does! Looks just fine.

Since you have access to the file now, I think we could probably implement any changes to make it netCDF-CF compliant. Just point out what needs to be fixed and I'm sure we can handle it over here.

If I can stay away from generating another python reader that'd be best.

Thanks!

[j-opatz]

Hello again,

I had some time to reexamine the configuration file you sent over and realized I had overlooked the solution: in your configuration file, you've misplaced the file_type parameter. It should be outside of the field entry, but inside the forecast dictionary. So similar to this:

fcst = {
file_type = NETCDF_NCCF;
field =
 { name = "cloud_area_fraction";
   level = "(0,0,*,*)";

...and so on. You'll also need to adjust the desc parameter, since whitespace isn't allowed. After these changes, your script should run as expected.

I did check your file to see if there were any outstanding CF compliance issues, and beyond the suggestion of adding a Conventions metadata section, your file is compliant.

Let me know if there's any other outstanding issues on this problem or if your issue is solved with the above steps!

[Wesley-J-Davis]

Thank you! Yes that worked perfectly. I'm sorry, it had been a while since I put together MODE config files, and it looks like I forgot how!