Evaluation, Reproducibility, Benchmarks Meeting 17

Minutes of meeting 17

Date: 22nd June 2022

Present: Carole, Michela, Jorge, Keyvan, Lena, Annika

WG Members will distribute the Metrics Reloaded survey among their labs Michela sent emails regarding the challenge data integration but did not get any replies yet Jorge will contact Nicola regarding metrics for memory consumption and runtime and will handle this point Points to discuss in Metrics Reloaded: Cloud computing

Annika and Carike prepared a table and check which metrics are already implemented in MONAI
Carole prepared issues and forwarded them to the MONAI dev team
- For example, metrics are mostly implemented for the binary case; multi-threshold metrics or metrics at a specific threshold need to be reworked/implemented
Testing process: Carole will propose them to use the pitfall figures as test cases for validating the implementations
Carole will think about the idea to publish a MONAI Metrics Reloaded Package

The Nature Methods editor really likes the use cases but is not convinced that it make sense to cover biology and medical domains in one framework
We will update Fig 12 of the paper (https://arxiv.org/abs/2206.01653)
Figure should convey the message that problems are very similar and lead to same recommendations (or only slight variations that are highlighted)
- Option: Add a new column with properties that are shared among examples (e.g. large organs)
- Option: One medical, one biological example per task
We will tweet and ask the community which use cases we should covor

Copyright (c) MONAI Consortium