Request help for Point_stat (METv10.0)

[harvir08]

Dear MET_HELP,

While running point_stat I didn't get any error message and it generated
some outfile as well but it has not created any stats. I have attached GTS
gts_rf_20210801.txt

observations in ascii and nc format along with one fcst file and point stat
config file. Please rectify the issue(s) and
let me know the issues.

command line information as follows:
point_stat
/home/umfcst/ShortJobs/MET/met9.1/data/INDIA/fcst/20210731/ncum_rain_20210731_d01_10x10.grib2
gts_rf_20210801.nc PointStatConfig_apcp1
DEBUG 1: Default Config File:
/home/ruchika/user-test/srinivas/9.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: PointStatConfig_apcp1
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073518265697
DEBUG 1: Forecast File:
/home/umfcst/ShortJobs/MET/met9.1/data/INDIA/fcst/20210731/ncum_rain_20210731_d01_10x10.grib2
DEBUG 1: Observation File: gts_rf_20210801.nc
DEBUG 2:
DEBUG 2:

DEBUG 2:
DEBUG 2: Reading data for APCP/A24.
DEBUG 2: Processing masking regions.
DEBUG 2: Processing geography data.
DEBUG 2: For APCP/A24 found 1 forecast levels, 0 climatology mean levels,
and 0 climatology standard deviation levels.
DEBUG 2:
DEBUG 2:

DEBUG 2:
DEBUG 2: Searching 651 observations from 505 messages.
DEBUG 2:
DEBUG 2:

DEBUG 2:
DEBUG 2: Processing APCP/A24 versus APCP/A24, for observation type ADPSFC,
over region FULL, for interpolation method NEAREST(1), using 0 matched
pairs.
DEBUG 2:
DEBUG 2:

DEBUG 2:
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V.stat
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_fho.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_ctc.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_cts.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_cnt.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_pct.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_pstd.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_pjc.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_prc.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_ecnt.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_rps.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_eclv.txt
DEBUG 1: Output file: ./point_stat_NCUM_240000L_20210801_000000V_mpr.txt

• [ ]

[jprestop]

HI @harvir08. Thank you for describing the problem you are experiencing. It would be helpful if I could run that case. Could you please post ncum_rain_20210731_d01_10x10.grib2, gts_rf_20210801.nc, and PointStatConfig_apcp1 to our FTP server following the instructions under "How to Send Us Data" in the Resources for Troubleshooting post so that I can have the data to run this case? Alternatively, you could run this case with a higher verbosity level by adding "-v 3" to the command line call. That should give you more information about why there are 0 matched pairs. Thank you!

[harvir08]

Hi,
Thank you for responding.
I have put the data in the following directory : /incoming/irap/met_help/singh_data as follows:

-rw-r--r-- 1 ftp ftp 66589 Sep 8 00:31 gts_rf_20210801.nc
-rw-r--r-- 1 ftp ftp 45509 Sep 8 00:31 gts_rf_20210801.txt
-rw-r--r-- 1 ftp ftp 66589 Sep 8 00:26 ls
-rw-r--r-- 1 ftp ftp 181368 Sep 8 00:30 ncum_rain_20210731_d01_10x10.grib2
-rw-r--r-- 1 ftp ftp 6314 Sep 8 00:28 PointStatConfig_apcp1

[jprestop]

Thank you for sending your data, @harvir08.

Thank you for sending up your data. I ran this case using a verbosity level of 3 by adding “-v 3” to the end of the command line call. In the log output, I could see that 651 observations were rejected due to the obs_var_name:

DEBUG 2: Processing APCP/A24 versus APCP/A24, for observation type ADPSFC, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs   = 0
DEBUG 2: Observations processed    = 651
DEBUG 2: Rejected: station id      = 0
DEBUG 2: Rejected: obs var name    = 651

Looking in the NetCDF file, I could see that the variable name was “APCP_A24”:

 obs_var =
  "APCP_A24" ;

I changed the name of the obs field name in your config file to be “APCP_A24” instead of “APCP”:

obs = {
   field = [
      {
        name       = "APCP_A24";
        level      = [ "A24" ];
        cat_thresh = [ >0.1 ];
      }
   ];
}

Rerunning the case, with the same verbosity level, I saw in the log output that 651 observations were now being rejected due to the valid_time:

DEBUG 2: Processing APCP/A24 versus APCP_A24/A24, for observation type ADPSFC, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 3: Number of matched pairs   = 0
DEBUG 3: Observations processed    = 651
DEBUG 3: Rejected: station id      = 0
DEBUG 3: Rejected: obs type        = 0
DEBUG 3: Rejected: valid time      = 651

I ran the following to find the valid time of the fcst file:

wgrib2 -vt  ncum_rain_20210731_d01_10x10.grib2
1:0:vt=2021080100

Looking in your NetCDF file, I could see the valid time for the observations was:

20210801_030000

In the config file, I increased the obs window from 5400 seconds to 10800 seconds:

obs_window = {
   beg = -10800;
   end =  10800;
}

Rerunning the case, with the same verbosity level, I saw in the log output that 651 observations were now being rejected due to a level mismatch:

DEBUG 2: Processing APCP/A24 versus APCP_A24/A24, for observation type ADPSFC, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs   = 0
DEBUG 2: Observations processed    = 651
DEBUG 2: Rejected: station id      = 0
DEBUG 2: Rejected: obs var name    = 0
DEBUG 2: Rejected: valid time      = 0
DEBUG 2: Rejected: bad obs value   = 0
DEBUG 2: Rejected: off the grid    = 0
DEBUG 2: Rejected: topography      = 0
DEBUG 2: Rejected: level mismatch  = 651

I took a look at the NetCDF/ASCII data in order to determine why the observations were being rejected due to a level mismatch. I noticed that there were some problems with the formatting of the ASCII data. The first column (message_type) and second column (station id) were essentially the same, with the first column representing the station id with a text value and the second representing the station id with a numerical value. The message type should be consistent across all observations, so I modified that value to be “GAUGE”. The ninth column (height) is intended to be in meters above sea level (msl) or above ground level (agl) of the observation value, but it appeared that the ASCII data had pressure values listed there, so I modified those to be 0, since measured precipitation is at the ground level. You can view detailed information about the ASCII2NC tool here:
https://met.readthedocs.io/en/main_v10.0/Users_Guide/reformat_point.html#ascii2nc-tool

I modified those values with the following command:

cat gts_rf_20210801.orig.txt | awk '{print "GAUGE",$2,$3,$4,$5,$6,$7,$8,0,$10,$11}' > gts_rf_20210801.gauge.txt

And then reran ASCII2NC:

/usr/local/met/bin/ascii2nc gts_rf_20210801.gauge.txt gts_rf_20210801.gauge.nc

I also modified your configuration file to add “message_type = [ "GAUGE" ];” and to change the level from “A24” to be “Z0” in order to specify the vertical level of the observation (see below). You can read more about the level values here:
https://met.readthedocs.io/en/main_v10.0/Users_Guide/config_options.html#settings-common-to-multiple-tools

obs = {
   message_type = [ "GAUGE" ];
   field = [
      {
        name       = "APCP_A24";
        level      = ["Z0"];
        cat_thresh = [ >0.1 ];
      }
   ];
}

I also added “is_precipation = TRUE;” to the configuration file (underneath “rank_corr_flag = TRUE;”) to ensure that rank values will include the cases where accum precip is 0 in the forecast AND 0 in the observations.

Rerunning Point-Stat with those changes, I now see that there are 651 matched pairs out of 651 observations:

DEBUG 2: Processing APCP/A24 versus APCP_A24/Z0, for observation type GAUGE, over region FULL, for interpolation method NEAREST(1), using 651 matched pairs.
DEBUG 3: Number of matched pairs   = 651
DEBUG 3: Observations processed    = 651

I have attached the modified ASCII file and the modified configuration file here for your reference. Please let us know if you have any other questions.

gts_rf_20210801.gauge.txt
PointStatConfig_gauge_Z0.txt

[harvir08]

Thank you
It's a great help.
Now it's working fine.

[jprestop]

Great! I'm glad it's working for you now. Thank you for letting us know.