Evaluation, Reproducibility, Benchmarks Meeting 16

Minutes of meeting 16

Date: 25th May 2022

Present: Carole, Michela, Jorge, Keyvan, Lena, Annika

Metrics Reloaded initiative - Metrics implementation:
- Annika and Carole will check which of the metrics are already implemented in MONAI and if we need some update
- Annika will send the prefinal metric table to Carole
- Carole will provide the metric table and results of the analysis to the Dev team
  - Contact person for multi-threshold and counting metrics: Paul Jäger
  - Contact person for distance-based metrics: Carole
- Within MONAI, the metrics should be grouped by metric families
MONAI Bundle (Configuration of models, including metadata, pt files, ..)
- Reproducibility is desired at different levels
  - Sharing weights/pretrained models
  - Sharing models that are ready to be used
  - Sharing models that are validated for a clinical application that are ready to be used
- We may have a simple Delphi process after summer
- We will discuss this topic in the next MICCAI SIG Board meeting (June 20th)
MONAI datasets
- The main work should be done by MONAI’s data working group
- Once they converged to a solution, we will make the connection to challenge organizers

Copyright (c) MONAI Consortium