Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2

Md. Rakibul Islam, Md. Zahid Hossain, Mustofa Ahmed, Most. Sharmin Sultana Samu

TL;DR

The study targets automated chest X-ray report generation and addresses challenges in producing detailed, clinically coherent radiology narratives. It systematically compares four multimodal architectures that fuse visual features from ViT-B16 or SWIN encoders with textual decoding via BART or GPT-2, trained end-to-end on the IU-Xray dataset with cross-attention grounding. SWIN-BART emerges as the strongest configuration across ROUGE, BLEU, and BERTScore, demonstrating robust text fidelity and clinical relevance, albeit with fixed-length outputs due to computational constraints. The work highlights the viability of vision-language models for radiology reporting while noting the need for longer outputs and more diverse datasets to enhance generalization and real-world deployment.

Abstract

Radiology plays a pivotal role in modern medicine due to its non-invasive diagnostic capabilities. However, the manual generation of unstructured medical reports is time consuming and prone to errors. It creates a significant bottleneck in clinical workflows. Despite advancements in AI-generated radiology reports, challenges remain in achieving detailed and accurate report generation. In this study we have evaluated different combinations of multimodal models that integrate Computer Vision and Natural Language Processing to generate comprehensive radiology reports. We employed a pretrained Vision Transformer (ViT-B16) and a SWIN Transformer as the image encoders. The BART and GPT-2 models serve as the textual decoders. We used Chest X-ray images and reports from the IU-Xray dataset to evaluate the usability of the SWIN Transformer-BART, SWIN Transformer-GPT-2, ViT-B16-BART and ViT-B16-GPT-2 models for report generation. We aimed at finding the best combination among the models. The SWIN-BART model performs as the best-performing model among the four models achieving remarkable results in almost all the evaluation metrics like ROUGE, BLEU and BERTScore.

Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2

TL;DR

The study targets automated chest X-ray report generation and addresses challenges in producing detailed, clinically coherent radiology narratives. It systematically compares four multimodal architectures that fuse visual features from ViT-B16 or SWIN encoders with textual decoding via BART or GPT-2, trained end-to-end on the IU-Xray dataset with cross-attention grounding. SWIN-BART emerges as the strongest configuration across ROUGE, BLEU, and BERTScore, demonstrating robust text fidelity and clinical relevance, albeit with fixed-length outputs due to computational constraints. The work highlights the viability of vision-language models for radiology reporting while noting the need for longer outputs and more diverse datasets to enhance generalization and real-world deployment.

Abstract

Radiology plays a pivotal role in modern medicine due to its non-invasive diagnostic capabilities. However, the manual generation of unstructured medical reports is time consuming and prone to errors. It creates a significant bottleneck in clinical workflows. Despite advancements in AI-generated radiology reports, challenges remain in achieving detailed and accurate report generation. In this study we have evaluated different combinations of multimodal models that integrate Computer Vision and Natural Language Processing to generate comprehensive radiology reports. We employed a pretrained Vision Transformer (ViT-B16) and a SWIN Transformer as the image encoders. The BART and GPT-2 models serve as the textual decoders. We used Chest X-ray images and reports from the IU-Xray dataset to evaluate the usability of the SWIN Transformer-BART, SWIN Transformer-GPT-2, ViT-B16-BART and ViT-B16-GPT-2 models for report generation. We aimed at finding the best combination among the models. The SWIN-BART model performs as the best-performing model among the four models achieving remarkable results in almost all the evaluation metrics like ROUGE, BLEU and BERTScore.
Paper Structure (9 sections, 11 figures, 4 tables)

This paper contains 9 sections, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Experimental Setup
  5. Dataset
  6. Data Preprocessing and Analysis
  7. Model Hyperparameters
  8. Result Analysis
  9. Conclusion and Future Work

Figures (11)

  • Figure 1: Proposed methodology for automated X-ray interpretation
  • Figure 2: Sample X-ray images and corresponding findings in the form of reports from the IU-Xray dataset. These reports are treated as the ground truth.
  • Figure 3: Distribution of report length in number of words
  • Figure 4: Report length distribution in train, test and validation split
  • Figure 5: Word cloud of reports before removing stopwords
  • ...and 6 more figures