Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation

Iustin Sîrbu, Iulia-Renata Sîrbu, Jasmina Bogojeska, Traian Rebedea

TL;DR

The paper addresses automated radiology report generation from chest X-ray images by proposing GIT-CXR, an end-to-end transformer augmented with multi-view inputs, contextual information, and a novel curriculum learning regime. It demonstrates that curriculum learning, especially for long reports, yields substantial gains in METEOR and F1-based clinical accuracy on the large MIMIC-CXR-JPG dataset, while matching standard NLG metrics like BLEU and ROUGE-L. The study provides extensive ablation and error analyses, showing that context, multi-view inputs, and classification guidance collectively improve performance, and that curriculum learning is a key driver for handling long-form, domain-specific reports. Its findings suggest practical potential for standardized, efficient radiology reporting, though it acknowledges limitations—most notably evaluation on a single dataset and the need for further work to mesh curriculum strategies with auxiliary classifiers for robust clinical deployment.

Abstract

Medical imaging is crucial for diagnosing, monitoring, and treating medical conditions. The medical reports of radiology images are the primary medium through which medical professionals attest their findings, but their writing is time consuming and requires specialized clinical expertise. The automated generation of radiography reports has thus the potential to improve and standardize patient care and significantly reduce clinicians workload. Through our work, we have designed and evaluated an end-to-end transformer-based method to generate accurate and factually complete radiology reports for X-ray images. Additionally, we are the first to introduce curriculum learning for end-to-end transformers in medical imaging and demonstrate its impact in obtaining improved performance. The experiments have been conducted using the MIMIC-CXR-JPG database, the largest available chest X-ray dataset. The results obtained are comparable with the current state-of-the-art on the natural language generation (NLG) metrics BLEU and ROUGE-L, while setting new state-of-the-art results on F1 examples-averaged, F1-macro and F1-micro metrics for clinical accuracy and on the METEOR metric widely used for NLG.

GIT-CXR: End-to-End Transformer for Chest X-Ray Report Generation

TL;DR

The paper addresses automated radiology report generation from chest X-ray images by proposing GIT-CXR, an end-to-end transformer augmented with multi-view inputs, contextual information, and a novel curriculum learning regime. It demonstrates that curriculum learning, especially for long reports, yields substantial gains in METEOR and F1-based clinical accuracy on the large MIMIC-CXR-JPG dataset, while matching standard NLG metrics like BLEU and ROUGE-L. The study provides extensive ablation and error analyses, showing that context, multi-view inputs, and classification guidance collectively improve performance, and that curriculum learning is a key driver for handling long-form, domain-specific reports. Its findings suggest practical potential for standardized, efficient radiology reporting, though it acknowledges limitations—most notably evaluation on a single dataset and the need for further work to mesh curriculum strategies with auxiliary classifiers for robust clinical deployment.

Abstract

Medical imaging is crucial for diagnosing, monitoring, and treating medical conditions. The medical reports of radiology images are the primary medium through which medical professionals attest their findings, but their writing is time consuming and requires specialized clinical expertise. The automated generation of radiography reports has thus the potential to improve and standardize patient care and significantly reduce clinicians workload. Through our work, we have designed and evaluated an end-to-end transformer-based method to generate accurate and factually complete radiology reports for X-ray images. Additionally, we are the first to introduce curriculum learning for end-to-end transformers in medical imaging and demonstrate its impact in obtaining improved performance. The experiments have been conducted using the MIMIC-CXR-JPG database, the largest available chest X-ray dataset. The results obtained are comparable with the current state-of-the-art on the natural language generation (NLG) metrics BLEU and ROUGE-L, while setting new state-of-the-art results on F1 examples-averaged, F1-macro and F1-micro metrics for clinical accuracy and on the METEOR metric widely used for NLG.
Paper Structure (30 sections, 1 equation, 4 figures, 4 tables)

This paper contains 30 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformers in Image Captioning.
  4. Radiology Report Generation
  5. Curriculum Learning
  6. Method
  7. GIT-CXR (SV)
  8. GIT-CXR (MV)
  9. GIT-CXR-CLS
  10. Context
  11. Curriculum Learning
  12. Experiments
  13. Dataset
  14. Evaluation Metrics
  15. Experimental Setup
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Example of a study of one patient.
  • Figure 2: Our proposed methods with their setups.
  • Figure 3: The metrics score evolution with report length.
  • Figure 4: Distributions for the length of the generated reports (blue) and the length of the target reports (yellow), which have been truncated to 192 tokens (orange vertical line).