-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Evaluation, Reproducibility, Benchmarks Meeting 10
AReinke edited this page Apr 28, 2021
·
1 revision
Date: 28th April 2021
Present: Lena, Annika, David, Jens, Nicola, Carole, Michela, Paul
- Michela presented update on the poll results:
- 6 challenges replied to the survey (M&Ms, Cataracts, PANDA, MSD, RibFrac, Rad-Path)
- Requirements for using data
- Ask users never try to de-anonymize data
- Log access requests to datasets
- Storage solution with enough bandwidth
- Follow licence
- Agree to all terms and conditions of challenges
- Licenses: CC-BY, CC-BY-SA, CC-BY-NC
- Users need to fill registration form in 67% of challenges => may be skipped for most challenges
- => All of them agreed that to have registration/license agreement in MONAI script and to enable direct access to the data using MONAI script
- 5/6 agree to store data at MONAI
- Most concerns regarding releasing test data
- Decision:
- Start with the data from the 6 challenges that replied
- Continue with MICCAI 2021 challenges later (most of them already released training data)
- Focus on challenges that last multiple years and data used regularly
- TODO: Define “registration” in terms of MICCAI challenges - is global registration possible?
- Michela will contact MONAI developer team and present our current status
- Delphi round 2:
- Two-stage mapping received strong support by expert panel
- Task mapping received strong support by expert panel
- Some inclusion criteria need to be refined (regression, video analysis, instance counting)
- Term “object detection” was confusing (localization is implied) => Will vote on the term in the next round (add expertise options “expert”, “medium”, …)
- Segmentation: Differentiate between volumes and contours
- Detection: Differentiate between types of localizing object (bounding boxes vs spheres etc) => Part of metric mapping
- Definition of metric mapping: Form groups of ~4 experts?