Our solution for the MedFM: Foundation Model Prompting for Medical Image Classification Challenge 2023

Authors: @Tianyi Wang^*, @Mengkang Lu^*, @Yong Xia^✉️

Institute: National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China

Roadmap

• Update the final solution: ✔️
• Upload the solution paper: ✔️
• Make the initial ReadMe: ✔️
• Document code: 🚧
• Release the source code: 🚧
• Update ReadMe: 🚧

Introduction

Please check our paper for more details.

Getting Started

Clone the repository

git clone https://github.com/TianyiFranklinWang/MedFM_Submission_Only.git

Setup environment

Python version: 3.8.17 (make sure to use compatible versions as other versions may encounter issues)

Make sure you are using Linux distributions as the codes are platform bundled

python -m pip install -r requirements.txt

You may encounter errors installing PyTorch please follow the official guide to install

Prepare data

• Download the file lists from the official repo and place them under the data folder.
• Create a folder named images under every sub-directory in MedFMC.
• Put all image files under images.
• The final structure should look like this:

  data
  └── MedFMC
      ├── chest
      │   ├── images
      │   │   └── ***.png
      │   └── ***.txt
      ├── colon
      │   ├── images
      │   │   └── ***.png
      │   └── ***.txt
      └── endo
          ├── images
          │   └── ***.png
          └── ***.txt

  Tips: Using ln -s TARGET LINK_NAME to create symbolic links may ease your work if you already have your data elsewhere.

Download model weight

• Download the open-sourced model weight from Google Drive.
• Create a pretrain folder and put the weight file underneath it.

The official weight file link will be revealed with the source code soon.

Train models

Use run.sh to train your models. The usage of the script is as follows:

./run.sh EXP_TYPE EXP_NUM SHOT_NUM DATA_TYPE

EXP_TYPE in 'no-ema_no-aug' 'no-ema_aug' 'ema_aug'
EXP_NUM in 1 2 3 4 5
SHOT_NUM in 1 5 10
DATA_TYPE in 'chest' 'colon' 'endo'

Note that for EXP_TYPE ema_aug, we only need to run 1-shot colon experiments. For the other two EXP_TYPE all combinations of SHOT_NUM and DATA_TYPE should be run.

The full list of experiments (95 in total) required is as follows:

./run.sh [no-ema_no-aug no-ema_aug] [1 2 3 4 5] [1 5 10] [chest endo colon]
./run.sh [ema_aug] [1 2 3 4 5] [1] [colon]

Inference on test set

Use test.sh to infer your models. The usage of the script is as follows:

./test.sh EXP_TYPE EXP_NUM SHOT_NUM DATA_TYPE DEVICE_ID

EXP_TYPE in 'no-ema_no-aug' 'no-ema_aug' 'ema_aug'
EXP_NUM in 1 2 3 4 5
SHOT_NUM in 1 5 10
DATA_TYPE in 'chest' 'colon' 'endo'
# DEVICE_ID is your cuda device id

The full list of inferences (95 in total) required is as follows:

./test.sh [no-ema_no-aug no-ema_aug] [1 2 3 4 5] [1 5 10] [chest endo colon] DEVICE_ID
./test.sh [ema_aug] [1 2 3 4 5] [1] [colon] DEVICE_ID

Ensemble results

There are two stages to form the final submission:

• Ensemble no-ema_no-aug results with no-ema_aug results.

  python ./tools/ensemble_no-ema_no-aug_with_aug.pyc

• Ensemble ema_aug results with the output from the previous stage.

  python ./tools/ensemble_no-ema_with_ema.pyc

Code Structure

If you wish to dive into the code, the source code will be available in the near future. Right now, the naming should be straightforward. The structure is as follows:

MedFM_Submission_Only
├── CITATION.cff                              # Configuration for citation
├── LICENSE                                   # License
├── README.md                                 # ReadMe
├── configs                                   # Configurations
├── docs
│   └── MedFM_Solution.pdf                    # Detailed solution paper
├── medfmc
│   └── models
│       ├── adapt_beit.pyc                    # Model with adapter
│       └── beit.pyc                          # Vanilla model
├── pyarmor_runtime_000000                    # Distribution related runtime
├── requirements.txt                          # Dependencies
├── run.sh                                    # Bash script for training
├── test.sh                                   # Bash script for testing
└── tools
    ├── ensemble_no-ema_no-aug_with_aug.pyc   # Ensemble no-ema_no-aug with no-ema-aug
    ├── ensemble_no-ema_with_ema.pyc          # Ensemble no-ema_aug with ema_aug
    ├── test_prediction.pyc                   # Vanilla test script
    ├── test_prediction_tta.pyc               # Test script with TTA
    └── train.pyc                             # Training script

License

This project is released under the Apache 2.0 license.

Citation

This citation is only for this repository.

BibTex

@software{Wang_MedFM_Submission_Only,
author = {Wang, Tianyi and Lu, Mengkang and Xia, Yong},
license = {Apache-2.0},
title = {{MedFM_Submission_Only}},
url = {https://github.com/TianyiFranklinWang/MedFM_Submission_Only}
}