Feature planning for MONAI core v1.3+

[wyli]

Hi Everyone,

This is to make a collection of 'raw' ideas towards major features for the upcoming release.
Please feel free to comment and we can prioritise accordingly:

• Bundle workflow Pythonic APIs and usability enhancements (Define python level abstraction for a Bundle #5821 interoperability monai label)
• Auto3DSeg accuracy benchmarking (documentation/tutorials) windows support/test?
• Digital pathology active learning workflows
• SAM fine-tuning with medial images (Segment Anything from META: Network implementation. Has someone started already? #6357)
• MONAI Label integration with SAM interactive workflows
• Transforms for spatial coordinates (point, bounding box, mesh) (Applying Transform Chain to Geometric Objects #4027)
• Transforms for different input spatial shapes, and resolutions (RandAffined does not work for differently sized images #3339, Extend/correct behaviour of RandAffined to allow for different input shapes #4491)
• Image IO using network sources (LoadImage and SaveImage using network source (S3 or Google Cloud Storage) #4135)
• TensorRT optimization for bundles (for example Support TensorRT export with dynamo #6577)
• Merging GenerativeModels components (https://github.com/Project-MONAI/GenerativeModels/tree/main/generative) into MONAI core
• Cookbook repo (Consolidating DICOM Best Practices, Techniques, and Libraries #6551 https://github.com/Project-MONAI/monai-dicom-utils/pull/14/files Developer's guide for memory management (Add tutorial on memory management tutorials#1435; tutorial?))
• Sliding window patch inference API extensions (https://github.com/Project-MONAI/MONAI/milestone/88)
• https://github.com/Project-MONAI/tutorials/tree/main/modules/interpretability interpretability module?

---

Finished:

• Pythonic APIs for model/bundle fine-tuning (Network Arch Code refactoring to allow replacing the last FC and add a freeze core parameter  #6552)
• bundle examples, task-specific demos, onnx, easy-to-debug, documentation example https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt
• Auto3DSeg decoupling execution from job creator (Auto3DSeg should separate bcprun as an unique execution method #6567, [WIP] 6054 Integrate azureml for autorunner training #6088)

[ericspod]

i'll reference bundle feedback #6647 here as well.

[aylward]

During the 2023.06.23 dev meeting, with strong support from Wenqi, Andy, Prerna, Eric, and others, the consensus was that MONAI would focus on Bundles for Release 1.3.

Notes from that meeting are here:

• https://docs.google.com/document/d/1q1OzI3LoWQe5byhrYrkoGQoz1h8tljEZYXv6KRHUjY4/edit?usp=sharing

Tasks from the above list that fall into the "Bundle" umbrella include:

• Bundle workflow Pythonic APIs and usability enhancements (Define python level abstraction for a Bundle #5821, examples, task-specific demos, onnx, interoperability monai label, easy-to-debug, documentation example https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt)
• Pythonic APIs for model/bundle fine-tuning (Network Arch Code refactoring to allow replacing the last FC and add a freeze core parameter Network Arch Code refactoring to allow replacing the last FC and add a freeze core parameter  #6552)
• SAM (bundle from Model Zoo) fine-tuning with medial images (Segment Anything from META: Network implementation. Has someone started already? #6357)
• MONAI Label integration with SAM (bundle from model zoo) interactive workflows
• TensorRT optimization for bundles (for example Support TensorRT export with dynamo #6577)

This is not a complete list, and there are tasks that will be accomplished that do not address bundles, but the majority of tasks for 1.3 should work towards making bundles easy to create, contribute, use, and refine.

What other bundle-related tasks should be considered?

1. Feedback about Monai bundle  #6647

[pdogra89]

June 30th key takeaways:
@ericspod : Simplify definitions and clear understanding of bundles in easy to consume tutorials.
@pdogra89 : Minimal Viable Bundle that does inference and fine-tuning. Ease of usability first priority and then Building bundles
@vikashg : Focus on usability of bundles to export bundles in pythonic paradigm.
@ericspod : Bundles beyond just MONAI core, to be explored. Design thinking; redefine scope and purpose of bundles.

[atbenmurray]

Lazy resampling roadmap is in discussion #7151

[vikashg]

nnUNet Vs TotalSegmentator Vs Auto3dSeg Documentation.

[wyli]

Lazy Resampling (https://github.com/Project-MONAI/discussions/7151)
Consider #4027 and #3339 in terms of Lazy Resampling

[function2-llx]

Hello, will there be a plan to make MONAI compatible with torch.compile?

[wyli]

thanks @function2-llx, we haven't started looking into the latest API yet, #6577 documents an initial feature request.

[ericspod]

I still think this could be looked at: Project-MONAI/MONAILabel#1549

[ericspod]

We want to get the cookbook repo going, I'm going to merge this change but then we could look at contributing to it.

[ericspod]

Perhaps graph neural networks and transforms for spatial data plus integration with Pytorch Geometric would be an idea.

[ericspod]

Refactoring, cleaning up, and reorganising the Tutorials repo should be done before merging the generative tutorials into it