Failing GridStat SEEPS testing with Beta6

[RachelNorth]

I'm testing the GridStat SEEPS capability in Beta6 and it's producing a FATAL ERROR (SEGFAULT) message. The MET command I'm using is:

grid_stat -v 4 regridded_prods_op_gl-mn_2021120100_f024_A24.nc regridded_gpm_2021120100_2021120200.nc ./METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir /path/to/output/dir

I've loaded the data files and the METplus configuration up onto the ftp server.

After having said I don't think it was loading the climatology file I re-ran with -v 9, and the output suggests it is reading (parts of?) the climatology file without a problem.

Example error log below.

Thanks!

11/14 15:54:41.886Z metplus.b6c82145 INFO: Running METplus v6.0.0-beta5 as user xxxx with command: /path/to/METplus_Dev/ush/run_metplus.py GridStat_global_24h_precip_SEEPS_testing.conf
11/14 15:54:41.888Z metplus.b6c82145 INFO: Log file: /path/to/METplus/testing/GridStat/logs/metplus.log.20241114155441
11/14 15:54:41.888Z metplus.b6c82145 INFO: METplus Base: /path/to/METplus-6.0.0-beta5
11/14 15:54:41.888Z metplus.b6c82145 INFO: Final Conf: /path/to/METplus/testing/GridStat/metplus_final.conf.20241114155441
11/14 15:54:41.888Z metplus.b6c82145 INFO: Config Input: /path/to/METplus-6.0.0-beta5/parm/metplus_config/defaults.conf
11/14 15:54:41.889Z metplus.b6c82145 INFO: Config Input: /path/to/config/GridStat_global_24h_precip_SEEPS_testing.conf
11/14 15:54:41.921Z metplus.b6c82145 INFO: Running wrapper: GridStat
11/14 15:54:41.923Z metplus.b6c82145 INFO: ****************************************
11/14 15:54:41.923Z metplus.b6c82145 INFO: * Running METplus GridStatWrapper
11/14 15:54:41.923Z metplus.b6c82145 INFO: *  at init time: 2021-12-01 00:00
11/14 15:54:41.923Z metplus.b6c82145 INFO: ****************************************
11/14 15:54:41.924Z metplus.b6c82145 INFO: Processing forecast lead 1 day
11/14 15:54:41.928Z metplus.b6c82145 INFO: COMMAND: /path/to/METBuild/MET12.0.0Beta6/bin/grid_stat -v 4 regridded_prods_op_gl-mn_2021120100_f024_A24.nc regridded_gpm_2021120100_2021120200.nc /path/to/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir {OUTPUT_BASE}/testing/GridStat/GridStat/seeps_test_external_regrid_NOV24/2021120100_024_um_24h
OUTPUT:
DEBUG 1: Start grid_stat by xxxx at 2024-11-14 15:54:44Z  cmd: /path/to/METBuild/MET12.0.0Beta6/bin/grid_stat -v 4 regridded_prods_op_gl-mn_2021120100_f024_A24.nc regridded_gpm_2021120100_2021120200.nc /path/to/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir {OUTPUT_BASE}/testing/GridStat/GridStat/seeps_test_external_regrid_NOV24/2021120100_024_um_24h 
DEBUG 2: OpenMP running on 1 thread(s).
DEBUG 1: Default Config File: /path/to/MET12.0.0Beta6/share/met/config/GridStatConfig_default
DEBUG 1: User Config File: /path/to/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcMet".
DEBUG 4: 
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4:      lat_ll: -49.875
DEBUG 4:      lon_ll: -0.125
DEBUG 4:   delta_lat: 0.25
DEBUG 4:   delta_lon: 0.25
DEBUG 4:        Nlat: 400
DEBUG 4:        Nlon: 1440
DEBUG 4: 
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcMet".
DEBUG 4: 
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4:      lat_ll: -49.875
DEBUG 4:      lon_ll: -0.125
DEBUG 4:   delta_lat: 0.25
DEBUG 4:   delta_lon: 0.25
DEBUG 4:        Nlat: 400
DEBUG 4:        Nlon: 1440
DEBUG 4: 
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcMet".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcMet".
DEBUG 3: Parsed set_attr_name = "PRECIP24"
DEBUG 3: Parsed set_attr_level = "A24"
DEBUG 3: Parsed is_precipitation = "true"
DEBUG 3: Parsed set_attr_level = "A24"
DEBUG 3: Parsed is_precipitation = "true"
DEBUG 4: ClimoCDFInfo::set_cdf_ta() -> Since "cdf_bins" = 1, no climatology CDF bins will be applied.
DEBUG 2: Using the forecast grid as the verification grid
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 1440 Ny: 400 lat_ll: -49.875 lon_ll: -0.125 delta_lat: 0.250 delta_lon: 0.250 wrapLon: true
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=1142765159
DEBUG 1: Forecast File: regridded_prods_op_gl-mn_2021120100_f024_A24.nc
DEBUG 1: Observation File: regridded_gpm_2021120100_2021120200.nc
DEBUG 4: 
DEBUG 4: Grid information:
DEBUG 4:    Projection: Lat/Lon
DEBUG 4:    Nx: 1440
DEBUG 4:    Ny: 400
DEBUG 4:    lat_ll: -49.875
DEBUG 4:    lon_ll: -0.125
DEBUG 4:    delta_lat: 0.250
DEBUG 4:    delta_lon: 0.250
DEBUG 4:    wrapLon: true
DEBUG 4: 
DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0
DEBUG 4:    plane max: 229.588
DEBUG 4:    valid time: 20211202_000000
DEBUG 4:    lead time: 240000
DEBUG 4:    init time: 20211201_000000
DEBUG 4:    accum time: 240000
DEBUG 4: 
DEBUG 3: Reading forecast data for precipitation_amount(*,*).
DEBUG 3: Applying conversion function.
DEBUG 4: 
DEBUG 4: Grid information:
DEBUG 4:    Projection: Lat/Lon
DEBUG 4:    Nx: 1440
DEBUG 4:    Ny: 400
DEBUG 4:    lat_ll: -49.875
DEBUG 4:    lon_ll: -0.125
DEBUG 4:    delta_lat: 0.250
DEBUG 4:    delta_lon: 0.250
DEBUG 4:    wrapLon: true
DEBUG 4: 
DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0
DEBUG 4:    plane max: 601.824
DEBUG 4:    valid time: 20211202_000000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 20211202_000000
DEBUG 4:    accum time: 240000
DEBUG 4: 
DEBUG 3: Reading observation data for precipitation_amount(*,*).
DEBUG 3: For precipitation_amount(*,*), found 0 forecast climatology mean and 0 standard deviation field(s), and 0 observation climatology mean and 0 standard deviation field(s).
DEBUG 2: Processing masking regions.
DEBUG 3: Processing grid mask: FULL
DEBUG 4: parse_grid_mask() -> parsing grid mask "FULL"
DEBUG 2: 
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2: 
DEBUG 2: Processing precipitation_amount(*,*) versus precipitation_amount(*,*), for smoothing method NEAREST(1), over region FULL, using 576000 matched pairs.
DEBUG 2: Computing Categorical Statistics.
FATAL ERROR (SEGFAULT): Process 38273 got signal 11 @ local time = 2024-11-14 15:54:54Z
FATAL ERROR (SEGFAULT): Look for a core file in /net/home/h02/fryl/NGVER/SEEPS/testing/GridStat
FATAL ERROR (SEGFAULT): Process command line: /data/users/cfver/METBuild/MET12.0.0Beta6/bin/grid_stat -v regridded_prods_op_gl-mn_2021120100_f024_A24.nc regridded_gpm_2021120100_2021120200.nc /path/to/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir {OUTPUT_DIR}/METplus/testing/GridStat/GridStat/seeps_test_external_regrid_NOV24/2021120100_024_um_24h 
11/14 15:54:55.041Z metplus.b6c82145 INFO: Finished running /path/to/METBuild/MET12.0.0Beta6/bin/grid_stat - took 0:00:13.019641
11/14 15:54:55.196Z metplus.b6c82145 (command_builder.py:262) ERROR: (command_builder.py:1332) Command returned a non-zero return code: /path/to/METBuild/MET12.0.0Beta6/bin/grid_stat -v 4 regridded_prods_op_gl-mn_2021120100_f024_A24.nc regridded_gpm_2021120100_2021120200.nc /path/to/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir {OUTPUT_DIR}/METplus/testing/GridStat/GridStat/seeps_test_external_regrid_NOV24/2021120100_024_um_24h
11/14 15:54:55.196Z metplus.b6c82145 INFO: Check the logfile for more information on why it failed: /path/to/METplus/testing/GridStat/logs/metplus.log.20241114155441
11/14 15:54:55.197Z metplus.b6c82145 (run_util.py:296) ERROR: GridStat had 1 error.
11/14 15:54:55.198Z metplus.b6c82145 INFO: Overwrite final conf here: /path/to/METplus/testing/GridStat/metplus_final.conf.20241114155441
11/14 15:54:55.200Z metplus.b6c82145 INFO: METplus took 0:00:14.200702 to run.
11/14 15:54:55.200Z metplus.b6c82145 (run_util.py:345) ERROR: METplus has finished running as user xxxx but had 1 error.
11/14 15:54:55.200Z metplus.b6c82145 INFO: Check the log file for more information: /path/to/METplus/testing/GridStat/logs/metplus.log.20241114155441

[mpm-meto]

We're not sure that the MET_SEEPS_GRID_CLIMO_NAME environment variable is now working. This worked in the version we debugged MET11rc2. But we can't find any evidence of being able to set the location of the climatology in the METplus wrapper and consequently can't see where MET reads it in either (or where from).

[hsoh-u]

The verbose level 7 shows the SEEPS climo file name.
The SEEPS climo name is from the config file

DEBUG 7: SeepsClimoBase::get_climo_filename() -> SEEPS climo name="/d1/personal/hsoh/data/SEEPS_climo/gridded/PPT24_seepsweights_grid.nc"

The verbose level 10 shows if it's from the environment variable.
The SEEPS climo name is from the environment variable:

DEBUG 10: get_env()  MET_SEEPS_GRID_CLIMO_NAME to /d1/personal/hsoh/data/SEEPS_climo/gridded/PPT24_seepsweights_grid.nc
DEBUG 7: SeepsClimoBase::get_climo_filename() -> SEEPS climo name="/d1/personal/hsoh/data/SEEPS_climo/gridded/PPT24_seepsweights_grid.nc"

export MET_SEEPS_GRID_CLIMO_NAME="/dev/shm/hsoh/PPT24_seepsweights_grid.nc"

DEBUG 10: get_env()  MET_SEEPS_GRID_CLIMO_NAME to /dev/shm/hsoh/PPT24_seepsweights_grid.nc
DEBUG 7: SeepsClimoBase::get_climo_filename() -> SEEPS climo name="/dev/shm/hsoh/PPT24_seepsweights_grid.nc"

[mpm-meto]

A bit more progress. We've installed rc1 and it gives the same SEGFAULT but with the debug 9 messages we can see that the SEEPS values look sensible and the SEEPS calcs actually complete. The failure after that.

DEBUG 9: compute_aggregated_seeps_grid() -> ix, iy, obs_cat, fcst_cat, seeps_score:1439 399 1 2 0.5562080741
DEBUG 9: compute_aggregated_seeps_grid() -> obs_sum, fcst_sum, seeps_score_partial_sum, category: 1892878.962 1833748.759 53.59060663 7
DEBUG 9: compute_aggregated_seeps_grid() -> dp_size, nan_count, bad_count: 576000 0 0
DEBUG 6: compute_aggregated_seeps_grid() -> SEEPS score=0.4894642879 score_wgt=0.352365001 pv1=0.3573611111 pv2=0.2624340278 pv3=0.1001041667 pf1=0.2178038194 pf2=0.3816979167 pf3=0.1203975694
DEBUG 7: compute_aggregated_seeps_grid() -> pvf = 0.186396 0.030472 0.000936 0.164493 0.187073 0.030132 0.006472 0.044889 0.069036 weight=1.736111111e-06 (1/576000)
FATAL ERROR (SEGFAULT): Process 24440 got signal 11 @ local time = 2024-11-15 15:09:45Z
FATAL ERROR (SEGFAULT): Look for a core file in /net/home/h02/fryl/NGVER/SEEPS/testing/GridStat
FATAL ERROR (SEGFAULT): Process command line: /data/users/cfver/METBuild/MET12.0.0rc1/bin/grid_stat -v 9 /data/users/fryl/SEEPS/METplus/testing/GridStat/regrid_data_plane/regridded_prods_op_gl-mn_2021120100_f024_A24.nc /data/users/fryl/SEEPS/METplus/testing/GridStat/regrid_data_plane/regridded_gpm_2021120100_2021120200.nc /spice/data/users/cfrd/MET_requirements/downloads/METplus-6.0.0-beta5/parm/met_config/GridStatConfig_wrapped -outdir /data/users/fryl/SEEPS/METplus/testing/MET12rc1/GridStat/GridStat/seeps_test_external_regrid_NOV24/2021120100_024_um_24h

[JohnHalleyGotway]

@mpm-meto and @RachelNorth, thank you for sending along this data. Just following up to confirm that I'm able to replicate this segfault on my end and will proceed with debugging it. I expect that it'll require a bugfix and will hope to do that prior to creating the MET-12.0.0 release in December. I'll follow up once I have more information to share.

[JohnHalleyGotway]

Good news, this is a very minor bug that for which there is an easy workaround and an easy fix... and its really unrelated to the actual SEEPS functionality.

The problem:

• When ONLY NetCDF SEEPS output is requested, Grid-Stat fails to initialize the NetCDF output file before writing to it... which leads to a segfault:

GRID_STAT_NC_PAIRS_FLAG_SEEPS = TRUE

The workaround:

• Request AT LEAST ONE additional NetCDF output type, such as writing the lat/lon field. I confirmed that the following settings prevent the segfault:

GRID_STAT_NC_PAIRS_FLAG_LATLON = TRUE
GRID_STAT_NC_PAIRS_FLAG_SEEPS = TRUE

The real fix:

• I'll provide a bugfix for MET-12.0.0 to patch the logic for Grid-Stat to check for SEEPS NetCDF output being requested and create the NetCDF matched pairs file prior to writing output to it.

@RachelNorth, the workaround described above should enable you to get back to testing out the actual SEEPS functionality in Grid-Stat. Please let us know what you find!

[JohnHalleyGotway]

Just noting that this bugfix is described in dtcenter/MET#3020 and fixed by pull requests dtcenter/MET#3021 and dtcenter/MET#3022. Thanks for pointing out this issue and providing data to make it easy to debug!