Ignite Ensemble training/evaluator for classification

[davidiommi]

Hello!

I saw your tutorial on ensemble training/evaluation for segmentation. I wanted to apply your cross-validation and ensemble training for classification on 3D images, but I get error messages during the validation handler. My problem is that I have to apply these two transforms from the "https://github.com/Project-MONAI/tutorials/blob/master/3d_classification/ignite/densenet_training_dict.py"

post_label = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
post_pred = Compose([EnsureType(), Activations(softmax=True)])

in the SupervisedTrainer, SupervisedEvaluator, and val_post_transforms (from the tutorial https://github.com/Project-MONAI/tutorials/blob/master/modules/cross_validation_models_ensemble.ipynb)

could you please show how you could modify the SupervisedTrainer and SupervisedEvaluator and val_post_transforms in case of classification and not segmentation?

Thanks in advance.

[Nic-Ma]

Hi @davidiommi ,

Thanks for your interest here.
Could you please help share your full test program and error message? I don't quite understand what's wrong in your case.
BTW, please provide your MONAI version.

Thanks in advance.

[davidiommi]

Thank you for your reply.

This is the chunk of code for training. I have two classes (0,1) for a dataset of 3D images. It is able to compute the loss function, but I have "INFO:ignite.engine.engine.SupervisedTrainer:Key metric: None best value: -1 at epoch: -1" When computing the metric. I am confused on what to write on the val_handlers and postprocessing to run correctly the code. I use the last stable version of Monai.

   loss_function = torch.nn.CrossEntropyLoss()
    optim = torch.optim.Adam(net.parameters(), opt.lr)

    val_post_transforms = Compose(
        [Activationsd(keys="pred", softmax=True), AsDiscreted(keys="pred", to_onehot=2)]
    )

    val_handlers = [

        CheckpointSaver(save_dir="./runs/", save_dict={"net": net}, save_key_metric=True),
    ]

    evaluator = SupervisedEvaluator(
        device=device,
        val_data_loader=val_loaders[index],
        network=net,
        inferer=SimpleInferer(),
        postprocessing=val_post_transforms,
        key_val_metric={"val_acc": Accuracy(output_transform=from_engine(["pred", "label"]))},
        additional_metrics={"val_ROCAUC": ROCAUC(output_transform=from_engine(["pred", "label"]))},
        amp=True if monai.utils.get_torch_version_tuple() >= (1, 6) else False,
        val_handlers=val_handlers
    )

    train_handlers = [
        ValidationHandler(validator=evaluator, interval=10, epoch_level=True),
        StatsHandler(tag_name="train_loss", output_transform=from_engine(["loss"], first=True)),
        CheckpointSaver(save_dir="./runs/", save_dict={"net": net, "opt": optim}, save_final=False),
    ]

    trainer = SupervisedTrainer(
        device=device,
        max_epochs=opt.epochs,
        train_data_loader=train_loaders[index],
        network=net,
        optimizer=optim,
        loss_function=loss_function,
        postprocessing=val_post_transforms,
        inferer=SimpleInferer(),
        additional_metrics={"train_acc": Accuracy(output_transform=from_engine(["pred", "label"]))},
        amp=True if monai.utils.get_torch_version_tuple() >= (1, 6) else False,
        train_handlers=train_handlers,
    )
    trainer.run()
    return net

models = [train(i) for i in range(opt.models_ensemble)]

[Nic-Ma]

Could you please help share your full test log?
I didn't see anything wrong in your code..
CC @wyli @ericspod can you find anything wrong here?

Thanks in advance.

[ericspod]

Perhaps for your validation post transforms you shouldn't have AsDiscreted(keys="pred", to_onehot=2). It's not immediately clear from other tutorials using the engines and classification what to do, but lookint at this one there is y_pred_trans which applies the activation only to the prediction, and y_trans which applies one-hot to the ground truth. The salient code is this where the auc_metric is calculated directly (assuming y is the ground truth batch and y_pred the predictions):

y_onehot = [y_trans(i) for i in decollate_batch(y)]
y_pred_act = [y_pred_trans(i) for i in decollate_batch(y_pred)]
auc_metric(y_pred_act, y_onehot)
result = auc_metric.aggregate()
auc_metric.reset()