Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation

Mingzheng Zhang, Jinfeng Gao, Dan Xu, Jiangrui Yu, Yuhan Qiao, Lan Chen, Jin Tang, Xiao Wang

TL;DR

The paper tackles X-ray report generation by moving beyond LLM-centric pipelines to leverage a pre-trained Mamba backbone with parameter-efficient fine-tuning. It introduces EMRRG, which combines an Efficient-Tuning SSM (ET-SSM) Mamba backbone and a LLM with a hybrid decoder that uses cross-attention and dynamic gating for end-to-end report generation. Through Partial LoRA on intermediate Mamba features and LoRA on the input projection, EMRRG achieves strong results on IU X-ray, MIMIC-CXR, and CheXpert Plus while updating only about 2.3% of parameters, boosting efficiency. This work demonstrates that non-Transformer vision backbones can be effectively integrated with LLMs for medical report generation, offering practical gains in both performance and computational cost, and sets a path for future exploration with larger models and broader fine-tuning strategies.

Abstract

X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence that can significantly reduce diagnostic burdens for clinicians and patient wait times. Existing MRG models predominantly rely on Large Language Models (LLMs) to improve report generation, with limited exploration of pre-trained vision foundation models or advanced fine-tuning techniques. Mainstream frameworks either avoid fine-tuning or utilize simplistic methods like LoRA, often neglecting the potential of enhancing cross-attention mechanisms. Additionally, while Transformer-based models dominate vision-language tasks, non-Transformer architectures, such as the Mamba network, remain underexplored for medical report generation, presenting a promising avenue for future research. In this paper, we propose EMRRG, a novel X-ray report generation framework that fine-tunes pre-trained Mamba networks using parameter-efficient methods. Specifically, X-ray images are divided into patches, tokenized, and processed by an SSM-based vision backbone for feature extraction, with Partial LoRA yielding optimal performance. An LLM with a hybrid decoder generates the medical report, enabling end-to-end training and achieving strong results on benchmark datasets. Extensive experiments on three widely used benchmark datasets fully validated the effectiveness of our proposed strategies for the X-ray MRG. The source code of this paper will be released on https://github.com/Event-AHU/Medical_Image_Analysis.

EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation

TL;DR

The paper tackles X-ray report generation by moving beyond LLM-centric pipelines to leverage a pre-trained Mamba backbone with parameter-efficient fine-tuning. It introduces EMRRG, which combines an Efficient-Tuning SSM (ET-SSM) Mamba backbone and a LLM with a hybrid decoder that uses cross-attention and dynamic gating for end-to-end report generation. Through Partial LoRA on intermediate Mamba features and LoRA on the input projection, EMRRG achieves strong results on IU X-ray, MIMIC-CXR, and CheXpert Plus while updating only about 2.3% of parameters, boosting efficiency. This work demonstrates that non-Transformer vision backbones can be effectively integrated with LLMs for medical report generation, offering practical gains in both performance and computational cost, and sets a path for future exploration with larger models and broader fine-tuning strategies.

Abstract

X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence that can significantly reduce diagnostic burdens for clinicians and patient wait times. Existing MRG models predominantly rely on Large Language Models (LLMs) to improve report generation, with limited exploration of pre-trained vision foundation models or advanced fine-tuning techniques. Mainstream frameworks either avoid fine-tuning or utilize simplistic methods like LoRA, often neglecting the potential of enhancing cross-attention mechanisms. Additionally, while Transformer-based models dominate vision-language tasks, non-Transformer architectures, such as the Mamba network, remain underexplored for medical report generation, presenting a promising avenue for future research. In this paper, we propose EMRRG, a novel X-ray report generation framework that fine-tunes pre-trained Mamba networks using parameter-efficient methods. Specifically, X-ray images are divided into patches, tokenized, and processed by an SSM-based vision backbone for feature extraction, with Partial LoRA yielding optimal performance. An LLM with a hybrid decoder generates the medical report, enabling end-to-end training and achieving strong results on benchmark datasets. Extensive experiments on three widely used benchmark datasets fully validated the effectiveness of our proposed strategies for the X-ray MRG. The source code of this paper will be released on https://github.com/Event-AHU/Medical_Image_Analysis.
Paper Structure (19 sections, 4 equations, 3 figures, 6 tables)

This paper contains 19 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mamba Network
  4. Parameter Efficient Fine-Tuning
  5. X-ray Report Generation
  6. Methodology
  7. Overview
  8. Input Encoding
  9. Efficient-Tuning SSM
  10. LLM with Hybrid Decoder
  11. Loss Function
  12. Experiments
  13. Datasets and Evaluation Metric
  14. Implementation Details
  15. Comparison on Public Benchmarks
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Comparison between (a). Existing ViT/Mamba-LLM framework for X-ray medical report generation; (b). Our newly proposed Efficient-Tuning based SSM for MRG.
  • Figure 2: An overview of our proposed X-ray medical report generation framework by efficient tuning Mamba networks, termed EMRRG.
  • Figure 3: Comparison of generated reports on the CheXpert dataset. Yellow highlights indicate sentences in the MambaXray-VL report matching the ground truth, cyan highlights denote matching sentences in our EMRRG report, and pink highlights represent matches shared by both models.