You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is a general confusion that could be possible when interpreting confidence intervals and quantiles. Consider a value that has a natural range between -100 and +100 (being, for example, a practical range for temperatures), which may be normalised (or standardised) and be considered instead in the normalised space.
The various thresholds and parameters to the scores being calculated could sometimes be either read as being in the natural space, or in the normalised/probability space. For example, consider https://scores.readthedocs.io/en/latest/api.html#scores.continuous.interval_score . It has gradually become clear to me that the score is in the natural range rather than normalised or probability range. q_u is the forecast value at the quantile level, and the under-prediction penalty is based on the natural range rather than the normalised range.
This may or may not be obvious to users of the score. It might be worth providing some clarifying information to the return statement, or or to the documentation. I may even have gotten this wrong. But I don't think it's immediately clear that the value (range) of the score will be sensitive to the domain (natural values) of the input rather than being expressed as a probability or deviation.
The API docs and tutorial could benefit from someone reading through and making sure that it's reasonably clear at all stages from this perspective.
The text was updated successfully, but these errors were encountered:
tennlee
changed the title
Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value
(low priority) Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value
Nov 14, 2024
There is a general confusion that could be possible when interpreting confidence intervals and quantiles. Consider a value that has a natural range between -100 and +100 (being, for example, a practical range for temperatures), which may be normalised (or standardised) and be considered instead in the normalised space.
The various thresholds and parameters to the scores being calculated could sometimes be either read as being in the natural space, or in the normalised/probability space. For example, consider https://scores.readthedocs.io/en/latest/api.html#scores.continuous.interval_score . It has gradually become clear to me that the score is in the natural range rather than normalised or probability range. q_u is the forecast value at the quantile level, and the under-prediction penalty is based on the natural range rather than the normalised range.
This may or may not be obvious to users of the score. It might be worth providing some clarifying information to the return statement, or or to the documentation. I may even have gotten this wrong. But I don't think it's immediately clear that the value (range) of the score will be sensitive to the domain (natural values) of the input rather than being expressed as a probability or deviation.
The API docs and tutorial could benefit from someone reading through and making sure that it's reasonably clear at all stages from this perspective.
The text was updated successfully, but these errors were encountered: