Research WG Kick off Meetings Summary

General Vision for MONAI Research

From our perspective, research focus in MONAI should aim to establish MONAI asTHE go-to platform for state-of-the-art Medical Deep Learning among end-users aswell as researchers of all levels. To this end, the Research Working group aims toguide the integration of new methodologies and tools into the MONAI framework byfocusing on community-driven progress and good scientific practice. We areconvinced that only by committing to these principles can we address the prevalentchallenges in our field such as the lack of standardized evaluation and reproducibility,thus establishing MONAI as a role model in the community and creating long-termimpact.

Structure of MONAI Research

It is great to see the examples and tutorial sections filling up with new contributions including multiple contributions from the community. As currently there are many different categories such as “example”, “user guide”, “workflow”, “tutorial”, “demo”, “research implementation” or “developer guide”, which are hosted in different places (wiki / google colab / main-repo/examples, main-repo/research), it might be helpful to start a discussion on how all these contributions could be structured and presented in the long-term in an intuitive and scalable fashion. To this end, the working group proposes a tree-like structure to arrive at different methodologies (e.g. “supervised learning” -> “semantic segmentation”). Each such methodology could then be filled with three types of contributions catering to the entire spectrum of target users of MONAI:

• Tutorials : This could be iPython notebooks or python scripts with minimal and well documented examples including contributions from the community. A potential task in this section for v0.3.0 could be to streamline the naming and location of current working examples.

• Research Paper Examples: As presented in v0.2.0, cutting-edge research papers utilizing the MONAI framework could be displayed as further exemplary application of MONAI. The promise of increased visibility and outreach provided by uploading work into MONAI could also incentivize researchers to use our tools in their future papers. However, as per the long-term vision outlined above, it might be advisable to control this process in the form of selection criteria. These could be based on yet-to-be decided scientific guidelines such as:

  • Proper Benchmarking: MONAI featured papers are required to benchmark contributions against the state-of-the-art in the respective task, preferably on more than one dataset.
  • Reproducibility: The utilized datasets should be publicly available, so as to ensure reproducibility of the results in follow-up studies
  • Community Relevance: The addressed work should be peer reviewed, relevant to the community and preferably support commonly available hardware setups.

• _ Baselines:_ Widely accepted state-of-the-art methods (such as e.g. the U-Net) could be benchmarked on multiple popular datasets and made available as “MONAI Baselines”. This could go hand in hand with general guidelines on how to adapt these baselines to a custom dataset for sound comparison or as state-of-the-art tools for end-users. In our opinion, this could attract more users to MONAI than adding re-implementations of older work with no performance guarantees to the Research Paper Examples s ection. This idea is closely tied to the plans of the Benchmarking-Working-Group, which is working on standardized datasets and evaluation schemes in MONAI. In a long-term vision this could also include a model zoo with stored parameters and/or online evaluation on the standardized MONAI datasets.

Contributions of the Research Working Group

The contributions of the working group largely depend on the feedback and the discussion we are now starting. Some initial ideas and efforts are:

• Starting to compile a vision, structure and scientific guidelines for MONAI Research.
• Exploring new methodologies and tools in MONAI: The working group compiled a list of possible methodological additions to MONAI and aims to explore such methods in form of prototype tutorials, so as to identify existing tools that can be integrated or missing components that should be implemented. As a pilot project, we generated a first mockup for object detection ( https://docs.google.com/document/d/1qLYG5r4wGFeZ_5ubMQfMUdwfDrbundNSGaaPjtDpnac/edit ) and filed a respective pull request at pytorch for missing components ( https://github.com/pytorch/vision/pull/2337 ).
• Coordination with the Benchmarking Group on how to implement our commitment for standardization and reproducibility in MONAI.