Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated tutorial notebook for interval score #736

Merged
merged 6 commits into from
Nov 13, 2024
Merged

Updated tutorial notebook for interval score #736

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c87f1bc
Updated tutorial for interval score
reza-armuei Nov 13, 2024
3bc1860
Update tutorials/Quantile_Interval_And_Interval_Scores.ipynb
tennlee Nov 13, 2024
11c0ac0
Update tutorials/Quantile_Interval_And_Interval_Scores.ipynb
tennlee Nov 13, 2024
3dd7286
Update tutorials/Quantile_Interval_And_Interval_Scores.ipynb
tennlee Nov 13, 2024
4d9a09a
Update tutorials/Quantile_Interval_And_Interval_Scores.ipynb
tennlee Nov 13, 2024
007b46d
Update tutorials/Quantile_Interval_And_Interval_Scores.ipynb
tennlee Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 31 additions & 20 deletions tutorials/Quantile_Interval_And_Interval_Scores.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,42 @@
"\n",
"The quantile interval and interval scores are consistent scoring functions to evaluate prediction intervals, where the aim is to predict an interval that captures uncertainty at a given confidence level. Quantile interval score can accept any values for the lower and upper quantile levels as long as $0 < \\text{lower quantile level} < \\text{upper quantile level} < 1$ is met, whereas interval score is only for forecast intervals with symmetric quantile level range. For example lower and upper quantile levels of 0.2 and 0.9 can be used by the quantile interval score but not by the interval score.\n",
"\n",
"The quantile interval score function is defined as sum of three penalties, as follows:\n",
"The quantile interval score function is defined as the sum of three penalties, as follows:\n",
"\n",
"- **Interval Width Penalty:**\n",
"$$ W = q_{u} - q_{l} $$\n",
"- **Over-Prediction Penalty:**\n",
"$$ U = \\begin{cases} \n",
"$$ O = \\begin{cases} \n",
"\\frac{1}{\\alpha_l} \\cdot (q_{l} - y) & \\text{if } y < q_{l} \\\\\n",
"0 & \\text{otherwise}\n",
"\\end{cases} $$\n",
"- **Under-Prediction Penalty:**\n",
"$$ O = \\begin{cases} \n",
"$$ U = \\begin{cases} \n",
"\\frac{1}{1 - \\alpha_u} \\cdot (q_{u} - y) & \\text{if } y > q_{u} \\\\\n",
"0 & \\text{otherwise}\n",
"\\end{cases} $$\n",
"So, the total score can be written as:\n",
"$$ S = W + U + O $$\n",
"**So, the total score can be written as:**\n",
"$$ S = W + O + U $$\n",
"\n",
"where $S$ is the scoring function (here *quantile interval score*), $q_l$ and $q_u$ are lower and upper quantile forecasts, and $y$ is observation, $\\alpha_l$ and $\\alpha_u$ are lower and upper quantile level. Lower values of the quantile interval scores are better. As you can see, this score penalizes the width of interval as well as the extent to which observation falls outside of the interval forecast (i.e., under-prediction penalty when observation is larger than upper-quantile forecast, and over-prediction penalty when observation is smaller than lower-quantile forecast).\n",
"where:\n",
"- $S$ is the scoring function (here quantile interval score),\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dotpoints are not rendering correctly in readthedocs. The screenshot below shows how they are currently rendering in readthedocs:
image

This can usually be fixed by adding an empty line before the first dot point.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tennlee fixed this post-merge (by adding an empty line before the first dot point).

tennlee marked this conversation as resolved.
Show resolved Hide resolved
"- $q_l$ is the lower quantile forecast,\n",
"- $q_u$ is the upper quantile forecast,\n",
"- $y$ is the observation,\n",
"- $\\alpha_l$ is the lower quantile level,\n",
"- $\\alpha_u$ is the upper quantile level.\n",
"\n",
"In the case of *interval score*, over- and under-prediction penalties can be simplified to:\n",
"Lower values of the quantile interval scores are better. As you can see, this score penalises the width of the interval as well as the extent to which the observations fall outside of the interval forecast (i.e., under-prediction penalty when the observations are larger than the upper-quantile forecasts, and over-prediction penalty when the observations are smaller than the lower-quantile forecasts).\n",
"\n",
"$$ U = \\begin{cases} \n",
"In the case of interval score, over- and under-prediction penalties can be simplified to:\n",
"\n",
"- **Over-Prediction Penalty:**\n",
"$$ O = \\begin{cases} \n",
"\\frac{2}{\\alpha} \\cdot (q_{l} - y) & \\text{if } y < q_{l} \\\\\n",
"0 & \\text{otherwise}\n",
"\\end{cases} $$\n",
"- **Under-Prediction Penalty:**\n",
"$$ O = \\begin{cases} \n",
"$$ U = \\begin{cases} \n",
"\\frac{2}{\\alpha} \\cdot (q_{u} - y) & \\text{if } y > q_{u} \\\\\n",
"0 & \\text{otherwise}\n",
"\\end{cases} $$\n",
Expand Down Expand Up @@ -83,11 +92,13 @@
"metadata": {},
"source": [
"In this example, we generate synthetic observation data and interval forecasts for temperature under two scenarios:\n",
"1. **Scenario 1**: There is a 50% chance that the temperature will be between 15 and 20 °C, with a lower quantile of 0.10 (indicating 10% cooler) and an upper quantile of 0.60 (indicating 40% warmer).\n",
"2. **Scenario 2**: There is a 50% chance that the temperature will be between 15 and 20 °C, with a lower quantile of 0.25 (indicating 25% cooler) and an upper quantile of 0.75 (indicating 25% warmer).\n",
"\n",
"**Scenario 1**: There is a 50% chance that the temperature will be between 15 and 20 °C, with a lower quantile of 0.10 (indicating 10% cooler) and an upper quantile of 0.60 (indicating 40% warmer).\n",
tennlee marked this conversation as resolved.
Show resolved Hide resolved
"\n",
"**Scenario 2**: There is a 50% chance that the temperature will be between 15 and 20 °C, with a lower quantile of 0.25 (indicating 25% cooler) and an upper quantile of 0.75 (indicating 25% warmer).\n",
tennlee marked this conversation as resolved.
Show resolved Hide resolved
"\n",
"\n",
"You might have already guessed why we chose these scenarios. Scenario 1 represents a case where the quantile range is not symmetric, so we can only use the *quantile interval score\" for this scenario. In contrast, Scenario 2 has a symmetric quantile range. allowing us to use both *quantile interval* and *interval scores*, which should yield the same results."
"You might have already guessed why we chose these scenarios. Scenario 1 represents a case where the quantile range is not symmetric, so we can only use the quantile interval score for this scenario. In contrast, Scenario 2 has a symmetric quantile range allowing us to use both the quantile interval score and the interval score, which should yield the same results."
]
},
{
Expand Down Expand Up @@ -138,7 +149,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's calculate scores for both scenarios. At first we will calculate the aggregated score. Later we will see how to calculate score for specific dimension(s) (e.g., for each time forecast step) by using `preserve_dims` or `reduce_dims` arguments."
"Now let's calculate scores for both scenarios. At first we will calculate the aggregated score. Later we will see how to calculate scores for specific dimension(s) (e.g., for each time forecast step) by using `preserve_dims` or `reduce_dims` arguments."
]
},
{
Expand Down Expand Up @@ -536,21 +547,21 @@
}
],
"source": [
"overal_qis_sc1 = quantile_interval_score(\n",
"overall_qis_sc1 = quantile_interval_score(\n",
" obs=obs_da,\n",
" fcst_lower_qtile=forecast_lower_da_1,\n",
" fcst_upper_qtile=forecast_upper_da_1,\n",
" lower_qtile_level=0.1,\n",
" upper_qtile_level=0.6\n",
")\n",
"overal_qis_sc1"
"overall_qis_sc1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As mentioned, we can use both the *quantile interval score* and the *interval score* for Scenario 2 due the symmetric quantile range of forecast intervals in this scenario. Both scores will yield the same results. Let's verify this now:"
"As mentioned, we can use either the *quantile interval score* or the *interval score* for Scenario 2 due to the symmetric quantile range of forecast intervals in this scenario. Both scores will yield the same results. Let's verify this now:"
tennlee marked this conversation as resolved.
Show resolved Hide resolved
]
},
{
Expand All @@ -560,7 +571,7 @@
"outputs": [],
"source": [
"# Quantile interval score for forecast Scenario 2\n",
"overal_qis_sc2 = quantile_interval_score(\n",
"overall_qis_sc2 = quantile_interval_score(\n",
" obs=obs_da,\n",
" fcst_lower_qtile=forecast_lower_da_2,\n",
" fcst_upper_qtile=forecast_upper_da_2,\n",
Expand Down Expand Up @@ -965,13 +976,13 @@
],
"source": [
"# Interval score for forecast Scenario 2\n",
"overal_is_sc2 = interval_score(\n",
"overall_is_sc2 = interval_score(\n",
" obs=obs_da,\n",
" fcst_lower_qtile=forecast_lower_da_2,\n",
" fcst_upper_qtile=forecast_upper_da_2,\n",
" interval_range=0.5,\n",
")\n",
"overal_is_sc2"
"overall_is_sc2"
]
},
{
Expand Down Expand Up @@ -1891,7 +1902,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Users have the option to input an array of weights within `quantile_interval_score` and `interval_score` to compute a weighted average score. An example scenario could involve assigning weights to regions based on their population when computing the these scores. The following is an example of using weights, with 1 mostly but a larger weight at a few locations based on (lat, lon)."
"Users have the option to input an array of weights within `quantile_interval_score` and `interval_score` to compute a weighted average score. An example scenario could involve assigning weights to regions based on their population when computing these scores. The following is an example of using weights, with 1 mostly but a larger weight at a few locations based on (lat, lon)."
]
},
{
Expand Down