METplus configuration variables in MET config files

[PerryShafran-NOAA]

Hi, there,

I've been told to generate a new discussion here. Currently I have METplus configuration files that work fine with the MET configuration files that work fine using METplus v4.0.0. However, I am told that it would be better for me to update my MET and METplus configuration files because of the change in the wrappers that allow me to remove the need to continue to maintain MET configuration files.

I had sent George McCabe some of my configuration file examples some time ago in the met_help email, before we moved into the GitHub question area, and he asked me to post the question here so it's no longer in the met_help email help. Thanks for your assistance!

Perry

[jprestop]

Hi @PerryShafran-NOAA. Thank you for your question. I understand that @georgemccabe is in the process of helping you with this issue and will follow up here as soon as he is able to.

[georgemccabe]

Thanks, Julie. @PerryShafran-NOAA , I will respond here with more information soon.

[JohnHalleyGotway]

@georgemccabe I'm wondering about the status of this discussion that's be open for a long time. Have you and @PerryShafran-NOAA coordinated offline? @PerryShafran-NOAA are you still waiting for resolution on this?

[georgemccabe]

@JohnHalleyGotway, this is still on my radar but it is a lower priority task since Perry's use cases still run as expected.

I have started going through some of @PerryShafran-NOAA's config files and updating them to include the new METplus configuration variables needed to use the generic wrapped MET config files. I have been taking notes of what I had to update and was planning on using this discussion as an example to outline instructions that users can take to update their use cases.

Perry, if you'd like I can send the updated version of one of the use case config files so you can use that to test that your use case behaves the same as before.

[PerryShafran-NOAA]

Yes I am still waiting, and have not coordinated with @georgemccabe offline. Thanks for reminding us of the open discussion.

[georgemccabe]

Hi @PerryShafran-NOAA,

I reviewed your CAM config files and have come up with a list of changes to produce the same results using the wrapped MET config files in the METplus repo instead of your _cam MET config files. The use case example you provided reads in many METplus config files that contain the same variables, so I also made changes to how the final conf files are written to separate out the config variables that are specific to the user's environment and runtime (available in 4.1.0 in a PR related to issue #1147). This allowed me to create a conf file with only the variables needed to run your use case. I have attached that file here (cam_updated.conf.txt) so you can test it out. You will only need to pass in this file and your system conf file to run_metplus.py. Let me know if you notice differences that shouldn't be there (Note there were some default values in the old versions of the wrapped MET config files pre v4.0.0 that differed from the MET defaults. The "incorrect" defaults in the files you copied to create your cam config files. I will outline these differences below).

Here is the list of the new METplus config variables I added:

PB2NC_LEVEL_RANGE_END = 255
PB2NC_QUALITY_MARK_THRESH = 9
PB2NC_TIME_SUMMARY_STEP = 3600
PB2NC_TIME_SUMMARY_WIDTH = 3600
PB2NC_MET_CONFIG_OVERRIDES = obs_bufr_map = [{ key = "HOVI";     val = "VIS"; },{ key = "TOCC";     val = "TCDC"; },{ key = "D_PBL";     val = "PBL";  },{ key = "D_CAPE";   val = "CAPE";  },{ key = "PMO";     val = "PRMSL";  },{ key = "MXGS";    val = "GUST";   },{ key = "TDO";     val = "DPT"; }];

POINT_STAT_MASK_POLY = "MET_BASE/poly/CONUS.poly", "MET_BASE/poly/NEC.poly","MET_BASE/poly/SEC.poly", "MET_BASE/poly/NWC.poly", "MET_BASE/poly/SWC.poly", "MET_BASE/poly/NMT.poly", "MET_BASE/poly/SMT.poly", "MET_BASE/poly/GRB.poly", "MET_BASE/poly/SWD.poly", "MET_BASE/poly/NPL.poly", "MET_BASE/poly/SPL.poly", "MET_BASE/poly/MDW.poly", "MET_BASE/poly/MDW.poly", "MET_BASE/poly/LMV.poly", "MET_BASE/poly/GMC.poly", "MET_BASE/poly/APL.poly"

POINT_STAT_OUTPUT_FLAG_FHO = STAT
POINT_STAT_OUTPUT_FLAG_CTC = STAT
POINT_STAT_OUTPUT_FLAG_SL1L2 = STAT
POINT_STAT_OUTPUT_FLAG_VL1L2 = STAT

I also changed the PB2NC_CONFIG_FILE and POINT_STAT_CONFIG_FILE values to point to the wrapped versions in the METplus repo. Starting in METplus 4.1.0 (beta2) these variables can simply be removed and the correct version will be used by default, but in 4.0.0 you need to explicitly set them:

PB2NC_CONFIG_FILE = {PARM_BASE}/met_config/PB2NCConfig_wrapped
POINT_STAT_CONFIG_FILE = {PARM_BASE}/met_config/PointStatConfig_wrapped

Below are details on how I determined what settings to add to the METplus config files.

Diffing MET Config Files

To find these variables I ran the diff command on your PB2NCConfig_cam file and the default config file found in the MET install location under share/met/config. I scanned the differences using a few rules. Left arrows (<) are from the default version and right arrows (>) are from the user version.

Settings to Ignore

• Variables found in the default version and not the user version can be ignored. They are likely new config variables that weren't available when the user version was created.

< message_type_group_map = [
<    { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";               },
<    { key = "ANYAIR";  val = "AIRCAR,AIRCFT";                      },
<    { key = "ANYSFC";  val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
<    { key = "ONLYSF";  val = "ADPSFC,SFCSHP";                      }
< ];

• Variables that differ that are controlled by an environment variable in the wrapped version can likely be ignored. If they were configurable via METplus when these files were created, then they will still be supported.

< message_type = [];
> //message_type = [];
> message_type = ${PB2NC_MESSAGE_TYPE} ;

Mismatched Default Values

• When a difference was found, I first checked the list of incorrect default values to see if the variable is included. While making updates for METplus 4.0.0, I discovered that there were some values set in the wrapped (in the METplus repo parm/met_config) versions that differ from the default (in MET install dir share/met/config) versions. This could lead to users inadvertently using settings that they didn't mean to use. The full list of these discrepancies can be found in the METplus User's Guide here: https://metplus.readthedocs.io/en/develop/Users_Guide/systemconfiguration.html#reconcile-default-values

Here is an example that I found: https://metplus.readthedocs.io/en/develop/Users_Guide/systemconfiguration.html#regrid-method-and-regrid-width
Your use case uses regrid method BILIN and width 2 in PointStat, while the default is NEAREST and 1. Leaving these out of the config file will use the correct default, but if BILIN and 2 is desired in your use case, you can add the settings listed in the User's Guide section:

POINT_STAT_REGRID_METHOD = BILIN
POINT_STAT_REGRID_WIDTH = 2

I also found that the pb_report_type in PB2NC was set to a list of values when the default is an empty list, which uses all reports. The old "incorrect" default was potentially filtering out data that you may want to use. Leaving it out of the METplus config will use the correct default, but if this filtering is still desired in your use case, you can set:

PB2NC_PB_REPORT_TYPE = 120, 220, 221, 122, 222, 223, 224, 131, 133, 233, 153, 156, 157, 180, 280, 181, 182, 281, 282, 183, 284, 187, 287

Differences that are configurable by METplus variables

83c78
<    end = 511;
---
>    end = 255;

This is a value that was changed from the default value. I searched for this line in the user config and saw that it is in the level_range dictionary. I navigated to the METplus User's Guide Python Wrappers PB2NC MET Configuration section and searched for the variable name to see if it is supported by the METplus wrappers. I found that the variable PB2NC_LEVEL_RANGE_END sets the level_range.end value, so I added it to the METplus config file:

PB2NC_LEVEL_RANGE_END = 255

I did the same for quality_mark_thresh, time_summary.step, and time_summary.width.

Differences that are not supported by METplus variables

• I noticed that the value for obs_bufr_map had been changed from the default value. I searched but did not find that the variable was supported via a METplus variable. I made a note to eventually add support for the variable in METplus issue Add support for commonly changed MET config variables part 2 #896. In the meantime, the MET config overrides variables can be used to set the value. This is essentially a catch-all where you can add text that will be added to the wrapped MET config file verbatim to override unsupported variables. See
  https://metplus.readthedocs.io/en/develop/Users_Guide/systemconfiguration.html#overriding-unsupported-met-configuration-variables for more information.

PB2NC_MET_CONFIG_OVERRIDES = obs_bufr_map = [{ key = "HOVI";     val = "VIS"; },{ key = "TOCC";     val = "TCDC"; },{ key = "D_PBL";     val = "PBL";  },{ key = "D_CAPE";   val = "CAPE";  },{ key = "PMO";     val = "PRMSL";  },{ key = "MXGS";    val = "GUST";   },{ key = "TDO";     val = "DPT"; }];

Combining multiple METplus config files

The changes I made as part of issue #1147 (PR coming soon) separate the variables written to the final conf file that are specific to a use case from the runtime settings (such as CLOCK_TIME, all log settings, MET_INSTALL_DIR, INPUT/OUTPUT_BASE, etc.). This allows you to run METplus with a list of config files and the final config file that is generated (controlled by METPLUS_CONF but typically written to {OUTPUT_BASE}/metplus_final.conf) and easily pull out the settings that were actually used for the use case that do not pertain to the user's environment. The runtime variables are moved to the [runtime] section so they can still be reviewed for context.