Updated tutorial notebook for interval score #736

reza-armuei · 2024-11-13T02:55:46Z

Please work through the following checklists. Delete anything that isn't relevant.

Development for new xarray-based metrics

Works with n-dimensional data and includes reduce_dims, preserve_dims, and weights args.
Typehints added
Add error handling
Imported into the API
Works with both xr.DataArrays and xr.Datasets if possible

Docstrings

Docstrings complete and follow Napoleon (google) style
Maths equation added
Reference to paper/webpage is in docstring. The preferred referencing style for journal articles is APA (7th edition)
Code example added

Testing of new xarray-based metrics

100% unit test coverage
Test that metric is compatible with dask.
Test that metrics work with inputs that contain NaNs
Test that broadcasting with xarray works
Test both reduce and preserve dims arguments work
Test that errors are raised as expected
Test that it works with both xr.DataArrays and xr.Datasets

Tutorial notebook

Short introduction to why you would use that metric and what it tells you
A link to a reference
A "things to try next" section at the end
Add notebook to Tutorial_Gallery.ipynb
Optional - a detailed discussion of how the metric works at the end of the notebook

Documentation

Add the score to the API documentation
Add the score to the included list of metrics and tools

Steph-Chong

Thanks @reza-armuei. Overall it looks good to me. I have made a few comments about some very minor things.

tutorials/Quantile_Interval_And_Interval_Scores.ipynb

Steph-Chong · 2024-11-13T10:14:04Z

tutorials/Quantile_Interval_And_Interval_Scores.ipynb

    "\n",
-    "where $S$ is the scoring function (here *quantile interval score*), $q_l$ and $q_u$ are lower and upper quantile forecasts, and $y$ is observation, $\\alpha_l$ and $\\alpha_u$ are lower and upper quantile level. Lower values of the quantile interval scores are better. As you can see, this score penalizes the width of interval as well as the extent to which observation falls outside of the interval forecast (i.e., under-prediction penalty when observation is larger than upper-quantile forecast, and over-prediction penalty when observation is smaller than lower-quantile forecast).\n",
+    "where:\n",
+    "- $S$ is the scoring function (here quantile interval score),\n",


The dotpoints are not rendering correctly in readthedocs. The screenshot below shows how they are currently rendering in readthedocs:

This can usually be fixed by adding an empty line before the first dot point.

@tennlee fixed this post-merge (by adding an empty line before the first dot point).

Co-authored-by: Stephanie Chong <[email protected]> Signed-off-by: Tennessee Leeuwenburg <[email protected]>

tutorials/Quantile_Interval_And_Interval_Scores.ipynb

Co-authored-by: Stephanie Chong <[email protected]> Signed-off-by: Tennessee Leeuwenburg <[email protected]>

tutorials/Quantile_Interval_And_Interval_Scores.ipynb

Co-authored-by: Stephanie Chong <[email protected]> Signed-off-by: Tennessee Leeuwenburg <[email protected]>

tennlee · 2024-11-13T10:47:37Z

The remaining issues relate to notebook rendering. It is easier for me to review and fix these post-merge, which is what I shall do. Thanks both.

Updated tutorial for interval score

c87f1bc

reza-armuei requested review from tennlee and Steph-Chong November 13, 2024 03:16

reza-armuei mentioned this pull request Nov 13, 2024

Quantile Interval and Interval Scores Tutorial #716

Closed