Calculating FSS for Ensemble Forecasts

[MReza-89]

Hi there,

I attempted to calculate the Fractions Skill Score (FSS) for ensemble forecasts of 24-hour accumulated rainfall using the following process:

• Utilised GenEnsProd to compute the Neighbourhood Ensemble Probability (NEP) for various event thresholds and neighbourhood window sizes.
• Used GridStat to read the NEP and observation fields and calculate the FSS.

However, the FSS results did not show the expected trend. I expected to see an increase in FSS values with larger window sizes, but instead, the FSS values decreased as the window size increased.
I should also mention that my observation grid has a coarser resolution compared to the model grid, and I set the observation grid as the verification grid. One question might be: “Should I perform the regridding in the GenEnsProd process or in GridStat?” I did an experiment, and both approaches yielded similar trends in FSS values. Interestingly, when I did not regrid to the observation grid (i.e., GRID_STAT_REGRID_TO_GRID = FCST) , I observed a reasonable trend in FSS values. Therefore, I am uncertain if the regridding process is influencing the results. I used both the NEAREST and BILIN regrid methods, and neither significantly impacted the results. Additionally, I suspected that this behavior might be due to how the grid points at the boundaries are handled. To check, I didanother experiment, changing GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH from 1 to 0.5. However, this adjustment did not significantly impact the results.

FYI, I performed another verification for deterministic model (no need to use GenEnsProd) with the same observation data, and this resulted in the expected FSS values.

I have shared some parts of my configuration settings below and would appreciate any insights or suggestions on what might be wrong or if there is anything that I should do differently.

GenEnsProd:

ENS_VAR1_NAME = APCP
ENS_VAR1_LEVELS = Z0
ENS_VAR1_THRESH = ge1.0, ge2.0

GEN_ENS_PROD_NBRHD_PROB_WIDTH = 1, 5, 11, 15, 21, 25
GEN_ENS_PROD_NBRHD_PROB_SHAPE = SQUARE
GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH = 1

GridStat:

FCST_GRID_STAT_VAR1_NAME = APCP_00_Z0_ENS_NEP_ge1.0_NBRHD1
FCST_GRID_STAT_VAR1_LEVELS = "(*,*)"
FCST_GRID_STAT_VAR1_THRESH = >=0.0

OBS_GRID_STAT_VAR1_NAME = apcp
OBS_GRID_STAT_VAR1_LEVELS = '(*,*)'
OBS_GRID_STAT_VAR1_THRESH = >=1.0
OBS_VAR1_OPTIONS = nbrhd = { field = OBS; shape = SQUARE; width = [ 1 ]; vld_thresh = 1.0; } output_flag = FALSE; output_flag = { nbrcnt = BOTH; }

FCST_GRID_STAT_VAR2_NAME = APCP_00_Z0_ENS_NEP_ge1.0_NBRHD25
FCST_GRID_STAT_VAR2_LEVELS = "(*,*)"
FCST_GRID_STAT_VAR2_THRESH = >=0.0

OBS_GRID_STAT_VAR2_NAME = apcp
OBS_GRID_STAT_VAR2_LEVELS = '(*,*)'
OBS_GRID_STAT_VAR2_THRESH = >=1.0
OBS_VAR2_OPTIONS = nbrhd = { field = OBS; shape = SQUARE; width = [ 5 ]; vld_thresh = 1.0; } output_flag = FALSE; output_flag = { nbrcnt = BOTH; }

Thanks,
Reza

[michelleharrold]

Hi Reza,

Thanks for your great discussion question! I have a couple of follow-up comments and questions, if that is okay, that might help us get on the right track.

1. First, @JohnHalleyGotway provided some excellent pro tips and comments regarding FSS for NEP (and NMEP) in this previous Discussion thread. This might provide some insight on how to optimally setup the configuration settings for the forecast and observation fields.

2. Are your forecast values being processed as scalars or probabilities? I do not see it being set in what you provided, but wanted to check.

3. What are the resolutions of your forecast and observation data? As you alluded to, there could be some sensitivities if there are large differences between the two.

Many thanks,
Michelle

[MReza-89]

Hi Michelle,

Thanks for your response.

To answer your questions/comments:

1. Yes, that was indeed a very useful discussion, and I followed John’s instructions in my work.

2. The data was processed as scalar, but based on the discussion you mentioned, this shouldn’t matter much. We are not calculating the fractional coverage for forecasts in the GridStat process since we already have it in the GenEnsProd output. Please correct me if I’m wrong, but GenEnsProd calculates the NEP (ranging from 0 to 1) of a forecast event exceeding a specified threshold. Therefore, we want GridStat to process the NEP fields as the forecast fractional coverage rather than calculating them itself. That’s why we need to set we need to set GRID_STAT_MET_CONFIG_OVERRIDES = nbrhd = { field = OBS; }.

3. The forecast has a 0.0198-degree resolution, while the observation has a 0.25-degree resolution. I suspected that the large difference in grid resolutions might cause some issues. However, I also tested our deterministic model with a grid resolution of 0.013 degrees using the same set of observations, and I observed reasonable results. The FSS trend behaved as expected when increasing the neighbourhood window size.

Please let me know if you need more information.

Thanks,
Reza

[MReza-89]

Hi there,

I attempted to calculate FSS for another model, our ensemble global model (ACCESS-GE), but this time against GPM data, and again I encountered the same issue. I used neighbourhood widths 1, 5, 11, 15, 21, 25, 31 and 41. The FSS value increases with the neighbourhood size from 1 to 21, but after that it drops. When I check Fractions Brier Score (FBS), I can see that it decreases (as expected) by increasing neighbourhood size from 1 to 41. So, it seems that something happening with FBS_ref

$$\text{FSS} = 1 - \frac{\text{FBS}}{\text{FBS}_{ref}}$$

FYI, previous examples were with the regional models (ACCESS-city and city ensemble).

Thanks,
Reza

[MReza-89]

Hi @michelleharrold and @JohnHalleyGotway,

May I follow-up on this discussion?

Thanks,
Reza