Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(low priority) Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value #737

Open
tennlee opened this issue Nov 13, 2024 · 0 comments
Assignees

Comments

@tennlee
Copy link
Collaborator

tennlee commented Nov 13, 2024

There is a general confusion that could be possible when interpreting confidence intervals and quantiles. Consider a value that has a natural range between -100 and +100 (being, for example, a practical range for temperatures), which may be normalised (or standardised) and be considered instead in the normalised space.

The various thresholds and parameters to the scores being calculated could sometimes be either read as being in the natural space, or in the normalised/probability space. For example, consider https://scores.readthedocs.io/en/latest/api.html#scores.continuous.interval_score . It has gradually become clear to me that the score is in the natural range rather than normalised or probability range. q_u is the forecast value at the quantile level, and the under-prediction penalty is based on the natural range rather than the normalised range.

This may or may not be obvious to users of the score. It might be worth providing some clarifying information to the return statement, or or to the documentation. I may even have gotten this wrong. But I don't think it's immediately clear that the value (range) of the score will be sensitive to the domain (natural values) of the input rather than being expressed as a probability or deviation.

The API docs and tutorial could benefit from someone reading through and making sure that it's reasonably clear at all stages from this perspective.

@tennlee tennlee changed the title Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value (low priority) Review interval score (and perhaps others) for clarity on what's normalised vs what's a natural value Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants