Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation

Trong Thang Pham, Ngoc-Vuong Ho, Nhat-Tan Bui, Thinh Phan, Patel Brijesh, Donald Adjeroh, Gianfranco Doretto, Anh Nguyen, Carol C. Wu, Hien Nguyen, Ngan Le

TL;DR

Fine-Grained CXR (FG-CXR) dataset is introduced, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy, and a novel explainable radiologist's attention generator network (Gen-XAI) is proposed that mimics the diagnosis process of radiologists.

Abstract

Developing an interpretable system for generating reports in chest X-ray (CXR) analysis is becoming increasingly crucial in Computer-aided Diagnosis (CAD) systems, enabling radiologists to comprehend the decisions made by these systems. Despite the growth of diverse datasets and methods focusing on report generation, there remains a notable gap in how closely these models' generated reports align with the interpretations of real radiologists. In this study, we tackle this challenge by initially introducing Fine-Grained CXR (FG-CXR) dataset, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy. Unlike existing datasets that include a raw sequence of gaze alongside a report, with significant misalignment between gaze location and report content, our FG-CXR dataset offers a more grained alignment between gaze attention and diagnosis transcript. Furthermore, our analysis reveals that simply applying black-box image captioning methods to generate reports cannot adequately explain which information in CXR is utilized and how long needs to attend to accurately generate reports. Consequently, we propose a novel explainable radiologist's attention generator network (Gen-XAI) that mimics the diagnosis process of radiologists, explicitly constraining its output to closely align with both radiologist's gaze attention and transcript. Finally, we perform extensive experiments to illustrate the effectiveness of our method. Our datasets and checkpoint is available at https://github.com/UARK-AICV/FG-CXR.

FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation

TL;DR

Fine-Grained CXR (FG-CXR) dataset is introduced, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy, and a novel explainable radiologist's attention generator network (Gen-XAI) is proposed that mimics the diagnosis process of radiologists.

Abstract

Developing an interpretable system for generating reports in chest X-ray (CXR) analysis is becoming increasingly crucial in Computer-aided Diagnosis (CAD) systems, enabling radiologists to comprehend the decisions made by these systems. Despite the growth of diverse datasets and methods focusing on report generation, there remains a notable gap in how closely these models' generated reports align with the interpretations of real radiologists. In this study, we tackle this challenge by initially introducing Fine-Grained CXR (FG-CXR) dataset, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy. Unlike existing datasets that include a raw sequence of gaze alongside a report, with significant misalignment between gaze location and report content, our FG-CXR dataset offers a more grained alignment between gaze attention and diagnosis transcript. Furthermore, our analysis reveals that simply applying black-box image captioning methods to generate reports cannot adequately explain which information in CXR is utilized and how long needs to attend to accurately generate reports. Consequently, we propose a novel explainable radiologist's attention generator network (Gen-XAI) that mimics the diagnosis process of radiologists, explicitly constraining its output to closely align with both radiologist's gaze attention and transcript. Finally, we perform extensive experiments to illustrate the effectiveness of our method. Our datasets and checkpoint is available at https://github.com/UARK-AICV/FG-CXR.

Paper Structure

This paper contains 22 sections, 1 equation, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Interpretable Deep Learning.
  4. Interpretable-oriented Datasets.
  5. Radiology Report Generation.
  6. Dataset: Fine-Grained CXR (FG-CXR)
  7. Anatomy Localization.
  8. Anatomical-aware Gaze Attention.
  9. Anatomical-aware Report
  10. Dataset Statistics
  11. How will our FG-CXR benefit the community?
  12. Methodology
  13. Gaze Attention Predictor (GAP)
  14. Spatial-aware Attended Encoder (SAP)
  15. Report Decoder
  16. ...and 7 more sections

Figures (5)

  • Figure 1: An overview of our interpretable Gen-XAI framework, generating a diagnosis report and its corresponding visual attention for each diagnosis in the report.
  • Figure 2: Annotation comparison between prior gaze-CXR datasets (left) which face challenges in aligning gaze location with textual description and our FG-CXR (right), given a CXR (middle).
  • Figure 3: Dataset distributions for FG-CXR.
  • Figure 4: The detailed architecture of our framework consisting of three key modules: Gaze Attention Predictor, Spatial-Aware Attended Encoder, and Report Decoder.
  • Figure 5: Qualitative comparison. The blue text highlights consistent content with ground truth. The red text indicates incorrect content.