Why/What
Describe impact during POC in terms of 'Future Technology', by delivering a Comparison Testbed which can be used to assess & quantify candidates for improvement. Examples:

• Forecasting algorithms used in Alliander/TenneT
• Forecasting algorithms used at Giga Storage
• Application/suitability of alternative forecasting algorithms

HOW:

1. Create testbed which takes an sklearn model as input and simple metrics as an output.
2. Use OpenSTEF to create benchmark. (relates to: https://github.com/alliander-opensource/AIFES/milestone/5 )
3. Pick a topic on which to compare a future technology (e.g. extreme values (preferred), or: forecasting in general, alternative pre-processing, other AI model, ...) (TenneT: currently mainly considers MAE, moving to RMSE. Sia partners; Consider the specific characteristics of the case study: so the maximum load)
   • ideas for evaluation of quantile forecasts
     • QQ-plot
     • realiability plot
     • fiability (OpenSTEF/openstef#404)

• Show that we are able to make a comparison to an alternative/novel/improved forecasting algorithm. (e.g. Notebook, example data, reference metrics). Links to: https://github.com/alliander-opensource/AIFES/milestone/12

Result:
Testbed for making comparison of improved forecasting versus baseline (Notebook + standard input data).

Involved parties: Sia Partners, Alliander, CWI, TUD (mostly advising on metrics)

update 2023-03-16;
Alliander has refined a story to share their current backtest framework and make is suitable for the Shared Data.