Different issues with Auto3DSeg and its notebook tutorial

[Maxime-Perret]

Hello everyone!

I'm opening this discussion to talk about the different issues I've encountered using the auto3dseg_autorunner_ref_api.ipynb notebook tutorial.

• The first issue I met was running the notebook without changing anything since the last update using the dataset for Task 09 (Spleen). I should first mention that my OS is Windows 11 Pro. When running algo.train(train_param) it fires commands including the files in the configs folder of each model, including network.yaml, network_search.yaml, transforms_infer.yaml, transforms_train.yaml, transforms_validate.yaml. Those files all cause issues because their path is not being interpreted as raw strings but instead the beginning of the file name gets changed into a newline for \n or a tab for \t. The error is then Invalid Argument (running with AutoRunner, the error is instead File not found), as shown below.
  
  I've personally fixed this issue with the following code snippet to rename the problematic files and update the commands accordingly, but I'm interested in a proper fix to solve this problem.

import os
# Rename problematic files on the computer containing \n or \t
# Add any prefix that doesn't start with a, b, f, n, r, t, v, ', ", ?, \
prefix = "m_"
for path, subdirs, files in os.walk(work_dir):
    for name in files:
        old = os.path.join(path, name)
        if os.path.basename(old).startswith(("network", "transform")):
            os.rename(old, old.replace(os.path.basename(old), prefix + os.path.basename(old)))

# Change history containing mention of files changed above
for task in history:
    for _, algo in task.items():
        algo.fill_records = dict((prefix + key, value) if key.startswith(("network", "transform")) \
                                    else (key, value) for (key, value) in algo.fill_records.items())

• Besides the bug above, the code then runs fine and produces results for DiNTS folds number 0, 1 and 2, but eventually the DiNTS fold n°3 runs out of memory as show below. The PC I'm using has 32GB of RAM.
  
  Is this amount of RAM really insufficient, or what else should I change? I also wanted to ask, I have been using PersistentDataset a lot for my MONAI models before Auto3DSeg, is there any way to use PersistentDataset with Auto3DSeg to reduce the workload for the Memory when applying the image transforms?

• Finally, I am curious about what does the modality field actually do in data_src_cfg in practice since I could not find where it actually gets used. And what about multi-modal image inputs? I am currently trying to replicate the multi-modal winning model of the HECKTOR challenge which uses Auto3DSeg with input being both CT and PET images. I observed the documentation mentions the DataAnalyzer provides no support for multi-modal images, but I could not make the Training part of the Auto3DSeg API work either.The documentation also mentions supporting CT as modality but not PET. I am curious how the aforementioned model was built with both the modalities, do you have any tips on how to reproduce it with Auto3DSeg?

Thanks a lot in advance for any help you can provide!

Best

[Nic-Ma]

Hi @mingxin-zheng ,

Could you please help take a look at this question?

Thanks in advance.

[mingxin-zheng]

Hi @Maxime-Perret ,

Thank you for the fix and bringing it up to the discussions!

• Would you be interested to submit your fix as a Pull Request? The algorithms templates are located at MONAI research contributions. Here is the overall guide about how to contribute and I am happy to assist if you have any question.
• The out-of-memory (OOM) issue can be a complicated one. My experiment showed the peak mem usage using GPU to train models is abot 20GB. I may be wrong here, but it seems your experiments are on CPU. Is that your plan to train the models on the CPU, or do you plan to use another hardware configuration somewhere else later?
• The "modality" (e.g. ct/mri) will have an effect in the algo.py in each algorithm. The algo.py is a script that would be called in the BundleGen (e.g. algorithm bundle generation) process. The effect is to apply different pre-process/data augmentation transforms.

[Maxime-Perret]

Hello @mingxin-zheng and thanks for your reply!

• I submitted a pull request. Let me know if there is anything else I should provide here or there.
• I saw your own pull request for the DataAnalyzer and will try to use those to see if it helps, I will let you know the results. In the meantime, I'm still curious if there's any way to use a similar system to PersistentDataset for Auto3DSeg.
• Thank you for the insight, I did not catch the use of modality. I am thus wondering how, according to NVAUTO team, the implementation was made using Auto3DSeg not only for PET images which is apparently not supported, but also for multi-modal inputs as I could not find any way to have more than one modality at a time.

[mingxin-zheng]

Thanks @Maxime-Perret !

• Thank you for submitting the PR. For topics around the backslash, let's discuss it under your PR then!
• The recent pull request of mine should fix some GPU mem issues and release GPU mem holding by PyTorch/CuPy calls. It sounds your issue is in memory rather than the GPU memory. Am I right?
• We have no explicit plan of using PersistentDatasets and multimodal images yet, but will work on the tutorials how to add customized algorithm scripts into the workflow. Would that be sufficient?

[Maxime-Perret]

Hello @mingxin-zheng

• I saw the answers in the PR, I'll answer there then
• As far as I understood, my issue was the DataAnalyzer processing the image transform on the CPU memory which went OOM. I thought the commit was supposed to have DataAnalzyer use the GPU memory instead, am I mistaken? Either way, I'll see if I can get it to work without running OOM with my hardware now.
• I understand, thank you. The NVAUTO team from Nvidia answered they'd make their implementation public in a bit where they used Auto3DSeg to treat multimodal images with a very large dataset, I'll wait until then to discover their secret!

Thanks a lot for your help!

[ShadowTwin41]

Dear @Maxime-Perret,

Did you find the implementation of NVAUTO for multimodal data?

Thank you!

[moonforsun]

Interested in any update on the multi-moddal images as input for the auto3dseg?

[Alisoltan82]

A really nice work @Maxime-Perret just faced the problem a day ago.

However, your solution gave me an error that :

algo is a string object with no fill_records function.

for your kind advice if anybody facing the same problem

[ShadowTwin41]

Dear all,

Is there any example of using auto3dseg for multimodal data? The tutorial provided in https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg only uses 1 modality as input.

Thank you!

[ShadowTwin41]

To the auto3dseg: https://github.com/Project-MONAI/tutorials/tree/main/auto3dseg
I want to apply it to the brats dataset.

[diazandr3s]

Hi @ShadowTwin41,

This is not a full tutorial, but you may find it useful for BRATS: https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10371

This is how the JSON should look like for BRATS: https://www.dropbox.com/s/b4dxpn81gwii0ut/brats23_gli_folds.json?dl=0

BTW, there is a PR to make the BRATS models available in the Auto3DSeg UI in Slicer: lassoan/SlicerMONAIAuto3DSeg#38

If you're interested in this, please consider making a comment.

[ShadowTwin41]

Hi @diazandr3s,

Thank you for your answer! It helped a lot!
I have another question, is it possible to run training on all the data at once instead of training on a k-fold configuration?

Thank you for your time!

[diazandr3s]

Yes. That's possible.
Either use the AutoRunner class and train on single fold like it is shown here (https://www.synapse.org/#!Synapse:syn51156910/discussion/threadId=10371) OR create two folds on the JSON file and train using YAML file - these two folds can actually be all the dataset repeated.

BTW, I'd suggest you use the SegResNet architecture only. It trains faster and it doesn't compromise performance. Something like this:

python -m monai.apps.auto3dseg AutoRunner run --input /workspace/dataset/configs/input.yaml --algos segresnet --work_dir /workspace/dataset/outputs/output_my_task

[ShadowTwin41]

Thank you for the tips! This tool looks very good, like the entire MONAI Project! Thank you very much for your work!

[ShadowTwin41]

I'm not sure if this is the best place to make a suggestion, but it would be interesting to think about a change to the training process, namely the way each epoch is counted. I think that in the current implementation each epoch corresponds to one iteration of all training data, which is the right definition for an epoch. However, for datasets with thousands of volumetric images, each epoch takes a very long time to train and store a checkpoint. I believe this was the reason that nnUNet defined each epoch as 200 iterations. Would it therefore be better to define each epoch as a predefined number of iterations, or do you think the current implementation is better/more efficient (due to the way MONAI data loading works)?