Evaluation, Reproducibility, Benchmarks Meeting 8

Minutes of meeting 8

Date: 24th February 2021

Present: Annika, Nicola, David, Jens, Lena, Carole, Paul, Keyvan

---

TOP 1: Delphi process on metrics

• General mapping approach was received well
• Overview mapping: Currently not captured that metric recommendations are influenced by clinical problem
• Task mapping
  • Make clear that one problem may be splitted into several intermediate problems (with different leaves and metrics)
  • Local vs global => Rephrase to “interested in spatial tasks”
• Add examples for the mappings in the paper
• Create a diagram of the delphi process for the paper
• Mention that we are not focussing on task-independent metrics, e.g. for assessing biases in data

---

TOP 2: Task forces reports

Easy data access task force (Michela)

• Annika will send email list of past organizers to Lena
• Start with MICCAI 2020 organizers

Metrics task force (Carole):

• Currently on hold

Benchmarking task force (Annika):

• Update on nondeterminism in deep learning systems
  • Pham et al. Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance, 2020
  • Bhojanapalli et al. On the Reproducibility of Neural Network Predictions, 2021
  • Nicola (or colleague) might give a talk on this topic in the DKFZ data science seminar