Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs

Xiaoman Zhang, Julián N. Acosta, Hong-Yu Zhou, Pranav Rajpurkar

TL;DR

This work introduces ReXKG, a knowledge-graph framework that converts radiology reports into structured graphs to evaluate AI-generated radiology reports beyond surface-level text. It defines six entity types and three relations, builds nodes and edges through UMLS mapping and semantic merging, and introduces three metrics—ReXKG-NSC, ReXKG-AMS, and ReXKG-SCS—for node similarity, edge distribution, and subgraph coverage. Through experiments on CheXpert Plus and MIMIC-CXR (with cross-modality demonstrations via CT-RATE and MIMIC-IV Head CT), the study shows generalist models extend entity and concept coverage more than specialists but still fall short of radiologist-level depth, particularly for devices and quantified measurements, and exhibit occasional longitudinal hallucinations. The results underscore the value of a KG-based evaluation to uncover knowledge gaps and guide development toward clinically aligned, longitudinal, multimodal radiology reporting systems.

Abstract

Recent advancements in artificial intelligence have significantly improved the automatic generation of radiology reports. However, existing evaluation methods fail to reveal the models' understanding of radiological images and their capacity to achieve human-level granularity in descriptions. To bridge this gap, we introduce a system, named ReXKG, which extracts structured information from processed reports to construct a comprehensive radiology knowledge graph. We then propose three metrics to evaluate the similarity of nodes (ReXKG-NSC), distribution of edges (ReXKG-AMS), and coverage of subgraphs (ReXKG-SCS) across various knowledge graphs. We conduct an in-depth comparative analysis of AI-generated and human-written radiology reports, assessing the performance of both specialist and generalist models. Our study provides a deeper understanding of the capabilities and limitations of current AI models in radiology report generation, offering valuable insights for improving model performance and clinical applicability.

Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs

TL;DR

This work introduces ReXKG, a knowledge-graph framework that converts radiology reports into structured graphs to evaluate AI-generated radiology reports beyond surface-level text. It defines six entity types and three relations, builds nodes and edges through UMLS mapping and semantic merging, and introduces three metrics—ReXKG-NSC, ReXKG-AMS, and ReXKG-SCS—for node similarity, edge distribution, and subgraph coverage. Through experiments on CheXpert Plus and MIMIC-CXR (with cross-modality demonstrations via CT-RATE and MIMIC-IV Head CT), the study shows generalist models extend entity and concept coverage more than specialists but still fall short of radiologist-level depth, particularly for devices and quantified measurements, and exhibit occasional longitudinal hallucinations. The results underscore the value of a KG-based evaluation to uncover knowledge gaps and guide development toward clinically aligned, longitudinal, multimodal radiology reporting systems.

Abstract

Recent advancements in artificial intelligence have significantly improved the automatic generation of radiology reports. However, existing evaluation methods fail to reveal the models' understanding of radiological images and their capacity to achieve human-level granularity in descriptions. To bridge this gap, we introduce a system, named ReXKG, which extracts structured information from processed reports to construct a comprehensive radiology knowledge graph. We then propose three metrics to evaluate the similarity of nodes (ReXKG-NSC), distribution of edges (ReXKG-AMS), and coverage of subgraphs (ReXKG-SCS) across various knowledge graphs. We conduct an in-depth comparative analysis of AI-generated and human-written radiology reports, assessing the performance of both specialist and generalist models. Our study provides a deeper understanding of the capabilities and limitations of current AI models in radiology report generation, offering valuable insights for improving model performance and clinical applicability.
Paper Structure (35 sections, 6 equations, 9 figures, 5 tables, 1 algorithm)

This paper contains 35 sections, 6 equations, 9 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Knowledge Graph Construction
  3. Information Extraction Schema
  4. Definition.
  5. Entity and Relation Extraction.
  6. Nodes Construction
  7. Edges Construction
  8. Knowledge Graph Evaluation Metrics
  9. Preliminary Definition
  10. KG Node Similarity Coefficient
  11. KG Adjacency Matrix Similarity
  12. KG Subgraph Coverage Score
  13. Experiments
  14. Datasets
  15. Experiments Settings
  16. ...and 20 more sections

Figures (9)

  • Figure 1: An illustration of Learning from Knowledge Graph.
  • Figure 2: Overview of the proposed knowledge graph construction system ReXKG. (a) The information extraction system for entity and relation extraction. (b) The node construction pipeline. (c) Illustration of edge construction.
  • Figure 3: Average count of concept entities used to modify disorders and anatomy across different models.
  • Figure 4: Detailed results of model predictions for given concepts related to specific disorders. Dark orange indicates the model predicts the relationship, while light orange indicates not.
  • Figure 5: Detailed results of whether the model predicts specific size measurements for given disorders. Dark orange indicates the model predicts, while light indicates not.
  • ...and 4 more figures