proposed codebase re structuring
update: proposal has been transformed into action items tracked via the following github issue: https://github.com/Project-MONAI/MONAI/issues/1015
origin: https://docs.google.com/document/d/1GY4n5vkbegRdRHufyzfkLjd1CXUfPY8KIslFsaY0eis
The Research Working group aims to guide the integration of new methodologies and tools into the MONAI framework by focusing on community-driven progress and good scientific practice. We are convinced that only by committing to these principles we can address the prevalent challenges in our field such as the lack of standardized evaluation and reproducibility, thus establishing MONAI as a role model in the community and creating long-term impact.
Before focusing on filling MONAI with methodologies and research content, the Research Working group identified an initial action item in establishing a process for systematic and concise categorisation and representation of methodologies in MONAI. Establishing a clear and systematic process for method implementation should help to scale research and methodological content in the future. Specifically, this proposal covers the following issues:
- Defining a Concise Set of Categories
- Consistent naming of and within categories
- General and consistent Entry-Points to method categories
- Scalable Folder-Structure within Categories.
- Selection Criteria for different categories.
It is great to see the examples and tutorial sections filling up with new items including multiple contributions from the community. Currently, there are many different categories hosted in different places:
Taken together, the current status of how methods are organised in MONAI features overlapping categories, ambiguous namings, and multiple entry points, which might be confusing for MONAI users. Further, the current folder structures within categories might be inconsistent or not scale in the long-term, when filled with more content.
To give an overview (more details below), the Research working group proposes the following structure and categories for representing methods in MONAI:
Following this proposal, all other namings (“Workflow”, “Demo”, “User Guide”) and entry points (e.g. selected links on the MONAI landing page) should be avoided for the sake of clarity. The proposed structure of methodologies in MONAI should help to distinguish and clearly categorize new content in MONAI.
-
Moving the tutorials into a dedicated repo was a great first step. Transparency could now be improved by mapping them to a dedicated subpage in the MONAI webpage via Sphynx-Gallery as outlined in https://github.com/pytorch/tutorials. This allows for a GUI-based and structured representation of tutorials such as in https://pytorch.org/tutorials and channels all previous links to one central entry point (i.e. a dedicated “Tutorials” section in the MONAI webpage).
-
Selection criteria in this category could be loose, allowing for community-based contributions (of e.g. showcasing external tools in conjunction with MONAI) and a broad range of topics and examples.
-
The folder structure could for the time being feature the following tree structure striving for scalability and intuitive user navigation:
- Supervised Learning
- Classification
- resnet
- 3D
- Ignite
- Dict (end-to-end examples)
- array
- Not ignite
- Ignite
- 2D
- densnet
- Semantic Segmentation
- Object Detection
- Instance Segmentation
- Semantic Regression (e.g. Registration)
- Weakly/Semi-Supervised Learning
- Unsupervised Learning
- Self-supervised Learning
- Generative Modelling
- Pipeline Tools / Module Examples (data augmentation examples, etc.)
-
Naming of tutorial instances should be made consistent avoiding terms such as “worfklow”, “demo”, or “test”.
-
Selective links for “User guides” and “workflows” on the MONAI landing page should be removed.
Looking ahead, “Research Examples” could be more focused on encouraging new research utilizing MONAI rather than spending resources on “MONAIfying” existing work. If existing work is deemed relevant enough for the community to be “MONAIfied” it could be considered as a core-component implementation in MONAI including educational tutorials.
Similar to tutorials, the research folder could be moved out of the main repo into a dedicated research-repo with associated links on the webpage (similar to the proposal for tutorials following https://pytorch.org/tutorials ).
The folder could for the time being feature the same tree structure as proposed for tutorials in 3.1 for the sake of consistency, scalability and intuitive user navigation
Selection criteria for integrated research should align with the MONAI core principles to support MONAI as a positive role model in the community :
- Proper Benchmarking of presented claims: MONAI featured papers should be required to benchmark contributions against the state-of-the-art in the respective task, preferably on more than one dataset.
- Reproducibility: The utilized datasets should be publicly available, so as to ensure reproducibility of the results in follow-up studies
- Community Relevance: The addressed work should be peer-reviewed, relevant to the community and preferably support commonly available hardware setups.
-
Looking further ahead: Widely accepted state-of-the-art methods (such as e.g. the U-Net) that are already implemented in the MONAI components could be benchmarked on multiple popular datasets and their scores, parameters and execution tutorials made available as “MONAI Baselines” accessed via an extra section on the MONAI webpage (similar to tutorials proposal).
-
This could go hand in hand with general guidelines on how to adapt these baselines to a custom dataset for sound comparison or as state-of-the-art tools for end-users. In our opinion, this could attract more users/researchers to MONAI than adding re-implementations of older work with no performance guarantees as “Research Examples”.
-
This idea is closely tied to the plans of the Benchmarking-Working-Group, which is working on standardized datasets and evaluation schemes in MONAI. In a long-term vision the resulting “Baseline-Repo” or “Model Zoo” could be presented on the webpage as open leaderboards for various MONAI-Datasets and the community could upload novel approaches for online evaluation using MONAI-Metrics.
-
The folder structure in MONAI components “networks/nets” could be assimilated to the tree structure proposed in 3.1 for scalability.
TODO: Discussion on who decides which Methods to implement in MONAI components / what are the selection criteria?
-
Discussion: What is the complementary value of the current folder “Examples” in the main-repo over “Tutorials” and “Research Examples” ? The working group argues that any new methodology in MONAI could be implemented in components, complemented by “Tutorials” for documentation and working examples and potentially complemented by latest “Research Examples” in MONAI-Research .
-
Thus, we propose to discard the example folder in the main-repo for the sake of streamlining and cleaning the structure of methodologies in MONAI.