Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset

Xiao Wang, Fuling Wang, Yuehang Li, Qingchuan Ma, Shiao Wang, Bo Jiang, Chuanfu Li, Jin Tang

TL;DR

A comprehensive benchmarking of existing mainstream X-ray report generation models and large language models, on the CheXpert Plus dataset is conducted and a large model for the X-ray image report generation is proposed using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning.

Abstract

X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence which can significantly reduce diagnostic burdens and patient wait times. Despite significant progress, we believe that the task has reached a bottleneck due to the limited benchmark datasets and the existing large models' insufficient capability enhancements in this specialized domain. Specifically, the recently released CheXpert Plus dataset lacks comparative evaluation algorithms and their results, providing only the dataset itself. This situation makes the training, evaluation, and comparison of subsequent algorithms challenging. Thus, we conduct a comprehensive benchmarking of existing mainstream X-ray report generation models and large language models (LLMs), on the CheXpert Plus dataset. We believe that the proposed benchmark can provide a solid comparative basis for subsequent algorithms and serve as a guide for researchers to quickly grasp the state-of-the-art models in this field. More importantly, we propose a large model for the X-ray image report generation using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning. Extensive experimental results indicate that the autoregressive pre-training based on Mamba effectively encodes X-ray images, and the image-text contrastive pre-training further aligns the feature spaces, achieving better experimental results. Source code can be found on \url{https://github.com/Event-AHU/Medical_Image_Analysis}.

CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset

TL;DR

A comprehensive benchmarking of existing mainstream X-ray report generation models and large language models, on the CheXpert Plus dataset is conducted and a large model for the X-ray image report generation is proposed using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning.

Abstract

X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence which can significantly reduce diagnostic burdens and patient wait times. Despite significant progress, we believe that the task has reached a bottleneck due to the limited benchmark datasets and the existing large models' insufficient capability enhancements in this specialized domain. Specifically, the recently released CheXpert Plus dataset lacks comparative evaluation algorithms and their results, providing only the dataset itself. This situation makes the training, evaluation, and comparison of subsequent algorithms challenging. Thus, we conduct a comprehensive benchmarking of existing mainstream X-ray report generation models and large language models (LLMs), on the CheXpert Plus dataset. We believe that the proposed benchmark can provide a solid comparative basis for subsequent algorithms and serve as a guide for researchers to quickly grasp the state-of-the-art models in this field. More importantly, we propose a large model for the X-ray image report generation using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning. Extensive experimental results indicate that the autoregressive pre-training based on Mamba effectively encodes X-ray images, and the image-text contrastive pre-training further aligns the feature spaces, achieving better experimental results. Source code can be found on \url{https://github.com/Event-AHU/Medical_Image_Analysis}.
Paper Structure (21 sections, 4 equations, 3 figures, 5 tables)

This paper contains 21 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. X-ray Medical Report Generation
  4. Pre-trained Large Models
  5. State Space Model
  6. MambaXray-VL Large Model
  7. Overview
  8. Multi-Stage Pre-training
  9. Implementation Details
  10. CXPMRG-Bench
  11. Mainstream MRG Algorithms
  12. LLMs for MRG
  13. Evaluation Results
  14. Experiments
  15. Dataset
  16. ...and 6 more sections

Figures (3)

  • Figure 1: An overview of the benchmarked LLM/VLM-based (green circle) and mainstream MRG models (blue circle) on the CheXpert Plus dataset in this paper.
  • Figure 2: An overview of our proposed MambaXray-VL pre-training framework. It contains three training stages, i.e., Mamba-based autoregressive generation, Xray-report based contrastive learning, and supervised fine-tuning. Specifically, the first phase mainly aims to make full use of larger-scale X-ray visual data to obtain a better visual backbone network (this paper chooses the low-complexity Mamba model). The second phase uses image-text contrastive loss to align X-ray images with medical reports. The third phase can fine-tune on various medical report generation datasets to obtain more refined X-ray report generation results. Note that the layers or modules with fire/snow symbols denote the parameters that are tuned/frozen in the training phase.
  • Figure 3: X-ray images and their corresponding ground-truths, along with the output of our model and R2GenGPT model generation reports on the MIMIC-CXR dataset. Matching sentences in our report are highlighted in yellow, R2GenGPT matching sentences are highlighted in cyan, and sentences matching by both models are highlighted in pink.