Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FITA: Fine-grained Image-Text Aligner for Radiology Report Generation

Honglong Yang, Hui Tang, Xiaomeng Li

TL;DR

The paper tackles the challenge of radiology report generation by enforcing fine-grained alignment between image patches and textual descriptions. It introduces FITA, a three-module framework consisting of the Image Feature Refiner (IFR), Text Feature Refiner (TFR), and Contrastive Aligner (CA) within a U-Transformer backbone. IFR uses Grad-CAM saliency maps and symptom supervision to refine image features, while TFR employs triplet-based refinement of textual representations guided by CheXBert-derived labels. CA applies an image–text contrastive loss to align the refined multimodal representations. Evaluated on MIMIC-CXR, FITA achieves state-of-the-art NLG performance and competitive clinical efficacy, with ablation studies showing the critical role of CA in maintaining alignment and boosting both linguistic quality and clinical accuracy.

Abstract

Radiology report generation aims to automatically generate detailed and coherent descriptive reports alongside radiology images. Previous work mainly focused on refining fine-grained image features or leveraging external knowledge. However, the precise alignment of fine-grained image features with corresponding text descriptions has not been considered. This paper presents a novel method called Fine-grained Image-Text Aligner (FITA) to construct fine-grained alignment for image and text features. It has three novel designs: Image Feature Refiner (IFR), Text Feature Refiner (TFR) and Contrastive Aligner (CA). IFR and TFR aim to learn fine-grained image and text features, respectively. We achieve this by leveraging saliency maps to effectively fuse symptoms with corresponding abnormal visual regions, and by utilizing a meticulously constructed triplet set for training. Finally, CA module aligns fine-grained image and text features using contrastive loss for precise alignment. Results show that our method surpasses existing methods on the widely used benchmark

FITA: Fine-grained Image-Text Aligner for Radiology Report Generation

TL;DR

The paper tackles the challenge of radiology report generation by enforcing fine-grained alignment between image patches and textual descriptions. It introduces FITA, a three-module framework consisting of the Image Feature Refiner (IFR), Text Feature Refiner (TFR), and Contrastive Aligner (CA) within a U-Transformer backbone. IFR uses Grad-CAM saliency maps and symptom supervision to refine image features, while TFR employs triplet-based refinement of textual representations guided by CheXBert-derived labels. CA applies an image–text contrastive loss to align the refined multimodal representations. Evaluated on MIMIC-CXR, FITA achieves state-of-the-art NLG performance and competitive clinical efficacy, with ablation studies showing the critical role of CA in maintaining alignment and boosting both linguistic quality and clinical accuracy.

Abstract

Radiology report generation aims to automatically generate detailed and coherent descriptive reports alongside radiology images. Previous work mainly focused on refining fine-grained image features or leveraging external knowledge. However, the precise alignment of fine-grained image features with corresponding text descriptions has not been considered. This paper presents a novel method called Fine-grained Image-Text Aligner (FITA) to construct fine-grained alignment for image and text features. It has three novel designs: Image Feature Refiner (IFR), Text Feature Refiner (TFR) and Contrastive Aligner (CA). IFR and TFR aim to learn fine-grained image and text features, respectively. We achieve this by leveraging saliency maps to effectively fuse symptoms with corresponding abnormal visual regions, and by utilizing a meticulously constructed triplet set for training. Finally, CA module aligns fine-grained image and text features using contrastive loss for precise alignment. Results show that our method surpasses existing methods on the widely used benchmark
Paper Structure (10 sections, 6 equations, 3 figures, 2 tables)

This paper contains 10 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. FITA: Fine-Grained Image-Text Aligner
  6. Training
  7. Experiments
  8. Experimental Settings
  9. Results
  10. Conclusion

Figures (3)

  • Figure 1: Left are two paired X-ray images and reports from MIMIC-CXR dataset. The right is the feature distribution for the pinpoint image patches and colored sentences. The circle denotes visual features and the rectangle denotes text features, with aligned visual and textual parts marking in the same color.
  • Figure 2: Illustration of our proposed Fine-grained Image-Text Aligner (FITA) approach, which adopts the U-Transformer structure and introduces three modules including Image Feature Refiner (IFR), Text Feature Refiner (TFR), Contrastive Aligner (CA).
  • Figure 3: Illustration of the image-report pair and its corresponding Grad-CAM for the observations. The colored boxes indicate the ground truth abnormal regions and the colored weights present the corresponding attention in the Grad-CAM.