Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brier score, quantile score descriptions #167

Open
wholmgren opened this issue Sep 15, 2020 · 0 comments
Open

brier score, quantile score descriptions #167

wholmgren opened this issue Sep 15, 2020 · 0 comments

Comments

@wholmgren
Copy link
Member

I was testing Quantile Skill Score in the reports and came across some results that initially confused me. Table below shows metrics for 4 pairs of forecasts

Forecast BS QS BSS QSS
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 0.0% 0.46 46.3 -0.113 0.653
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 40.0% 0.282 39.6 -0.161 0.688
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 50.0% 0.25 44.7 0.00e+00 0.643
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 100.0% 0.264 93.4 0.55 0.201
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 0.0% 0.415 129 nan nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 40.0% 0.243 122 nan nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 50.0% 0.25 121 nan nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 100.0% 0.585 112 nan nan

I came away from this wanting a few things in the brier score documentation

  1. a statement about how the Arbiter computes o for non-event forecasts (is f <= x?).
  2. some discussion about brier score vs. quantile score for quantile forecasts (like all of our reference forecasts as of today).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant