Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models

Yingshu Li, Zhanyu Wang, Yunyi Liu, Lei Wang, Lingqiao Liu, Luping Zhou

TL;DR

KARGEN is presented, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs that integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports.

Abstract

Harnessing the robust capabilities of Large Language Models (LLMs) for narrative generation, logical reasoning, and common-sense knowledge integration, this study delves into utilizing LLMs to enhance automated radiology report generation (R2Gen). Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. This is achieved by leveraging the knowledge graph to distill disease-related features in a designed way. Since a radiology report encompasses both normal and disease-related findings, the extracted graph-enhanced disease-related features are integrated with regional image features, attending to both aspects. We explore two fusion methods to automatically prioritize and select the most relevant features. The fused features are employed by LLM to generate reports that are more sensitive to diseases and of improved quality. Our approach demonstrates promising results on the MIMIC-CXR and IU-Xray datasets.

KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models

TL;DR

KARGEN is presented, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs that integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports.

Abstract

Harnessing the robust capabilities of Large Language Models (LLMs) for narrative generation, logical reasoning, and common-sense knowledge integration, this study delves into utilizing LLMs to enhance automated radiology report generation (R2Gen). Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. This is achieved by leveraging the knowledge graph to distill disease-related features in a designed way. Since a radiology report encompasses both normal and disease-related findings, the extracted graph-enhanced disease-related features are integrated with regional image features, attending to both aspects. We explore two fusion methods to automatically prioritize and select the most relevant features. The fused features are employed by LLM to generate reports that are more sensitive to diseases and of improved quality. Our approach demonstrates promising results on the MIMIC-CXR and IU-Xray datasets.
Paper Structure (8 sections, 5 equations, 3 figures, 3 tables)

This paper contains 8 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Feature Extraction
  4. Feature Fusion
  5. Report generation
  6. Experiments
  7. Conclusions
  8. Disclosure of Interests.

Figures (3)

  • Figure 1: An overview of the KARGEN framework, which comprises a visual encoder, a knowledge-enhanced encoder, a fusion module and a report generator.
  • Figure 2: The medical domain knowledge graph is constructed based on the correlations among various diseases, where diseases that are linked together are interconnected.
  • Figure 3: Examples of the generated reports. For better illustration, different colours highlight different medical terms in the reports.