[COMMSENG'24, TMI'24] Interactive Computer-Aided Diagnosis using LLMs

This repo includes official implementations of ChatCAD and ChatCAD+

Paper

[Nature COMMSENG] ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models by Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, Dinggang Shen

(a) Overview of our proposed strategy. The image is processed by various networks to generate diverse outputs, which are then transformed into text descriptions. The descriptions, served as a link between visual and linguistic information, are combined as inputs to a large language model (LLM). With its ability to reason and its knowledge of the medical field, the LLM can provide a condensed report. (b) Interactive explanations and medical advice from ChatCAD.

[IEEE TMI] ChatCAD+: Towards a Reliable and Universal Interactive CAD using LLMs by Zihao Zhao*, Sheng Wang*, Jinchen Gu*, Yitao Zhu*, Lanzhuju Mei, Zixu Zhuang, Zhiming Cui, Qian Wang, Dinggang Shen

Overview of our proposed ChatCAD+ system. (a) For patients seeking a diagnosis, ChatCAD+ generates reliable medical reports based on the input medical image(s) by referring to local report database. (b) Additionally, for any inquiry from patients, ChatCAD+ retrieves related knowledge from online database and lets large language model generate reliable response.

Introduction

This repository provides the official implementation of some components of ChatCAD+:

• Modality identification
• Chinese version Interactive CAD of Chest X-rays
• LLM-based knowledge retrieval
• An easy-deploy local web ui based on Gradio

Resources

• We would like to thank Merck Manual Professional who make all these medical knowledge public, we sorted their website for easier usage: here
• A BART-based model that has the capability to translate chest X-ray reports into Chinese well [link]

Usage

weights&others

• R2GenCMN: r2gcmn_mimic-cxr.pth and annotation.json
• PCAM weights: JFchexpert.pth
• Place annotation.json under ./r2g/ and pre-trained weights under ./weights/
• For template retrieval system, please download MIMIC-CXR reports from official website and organize them into a dictionary, save as report_en_dict.json under the ./

You can either find them from original repository or dowload from Google Drive

Deploy local web ui

• pip install -r requirements.txt
• implement web.py and load your openai api-key

- Would like some diagnostic results? upload image via left panel --> wait for your report

- ChatCAD+ will answer your question with a reference from Merck Manucal Professional

Citation

If you find our work useful, please consider giving a star ⭐ and citation.

@article{wang2023chatcad,
  title={Chatcad: Interactive computer-aided diagnosis on medical image using large language models},
  author={Wang, Sheng and Zhao, Zihao and Ouyang, Xi and Wang, Qian and Shen, Dinggang},
  journal={arXiv preprint arXiv:2302.07257},
  year={2023}
}

@article{zhao2023chatcad,
      title={ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs},
      author={Zihao Zhao and Sheng Wang and Jinchen Gu and Yitao Zhu and Lanzhuju Mei and Zixu Zhuang and Zhiming Cui and Qian Wang and Dinggang Shen},
      journal={arXiv preprint arXiv:2305.15964},
      year={2023},
}

Acknowledgment

Our implementation (including coming version) is based on the following codebases. We gratefully thank the authors for their wonderful works.

R2GenCMN, PCAM, CSNet.