Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ORID: Organ-Regional Information Driven Framework for Radiology Report Generation

Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai

TL;DR

This paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs on the radiology report generation.

Abstract

The objective of Radiology Report Generation (RRG) is to automatically generate coherent textual analyses of diseases based on radiological images, thereby alleviating the workload of radiologists. Current AI-based methods for RRG primarily focus on modifications to the encoder-decoder model architecture. To advance these approaches, this paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs. Specifically, based on the LLaVA-Med, we first construct an RRG-related instruction dataset to improve organ-regional diagnosis description ability and get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion module to effectively combine the information from the organ-regional diagnosis description and radiology image. To further reduce the influence of noise from unrelated organs on the radiology report generation, we introduce an organ importance coefficient analysis module, which leverages Graph Neural Network (GNN) to examine the interconnections of the cross-modal information of each organ region. Extensive experiments an1d comparisons with state-of-the-art methods across various evaluation metrics demonstrate the superior performance of our proposed method.

ORID: Organ-Regional Information Driven Framework for Radiology Report Generation

TL;DR

This paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs on the radiology report generation.

Abstract

The objective of Radiology Report Generation (RRG) is to automatically generate coherent textual analyses of diseases based on radiological images, thereby alleviating the workload of radiologists. Current AI-based methods for RRG primarily focus on modifications to the encoder-decoder model architecture. To advance these approaches, this paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs. Specifically, based on the LLaVA-Med, we first construct an RRG-related instruction dataset to improve organ-regional diagnosis description ability and get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion module to effectively combine the information from the organ-regional diagnosis description and radiology image. To further reduce the influence of noise from unrelated organs on the radiology report generation, we introduce an organ importance coefficient analysis module, which leverages Graph Neural Network (GNN) to examine the interconnections of the cross-modal information of each organ region. Extensive experiments an1d comparisons with state-of-the-art methods across various evaluation metrics demonstrate the superior performance of our proposed method.

Paper Structure

This paper contains 33 sections, 9 equations, 16 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Image Captioning
  4. Radiology Report Generation
  5. Multimodal Large Language Model
  6. Method
  7. LLaVA-Med-RRG
  8. Organ-based Cross-modal Fusion Module
  9. Data Flow.
  10. Modality Fusion.
  11. Organ Importance Coefficient Analysis Module
  12. Radiology Report Generation Module
  13. Experiment Settings
  14. Datasets
  15. Baseline and Evaluation Metrics
  16. ...and 18 more sections

Figures (16)

  • Figure 1: Visualization of organ-regional radiology image and diagnosis descriptions. Relevant segments associated with the target report have been highlighted using distinct colors.
  • Figure 2: The overall architecture of our proposed ORID framework.
  • Figure 3: Input and output type during the instruction tuning.
  • Figure 4: An example of LLaVA-Med's organ-reional diagnosis description compare with that of LLaVA-Med-RRG. The sentences that are correct or highly-related with target reports have been marked in green, otherwise have been marked in red.
  • Figure 5: Statistical analysis of question-answer pairs and average token length for each organ.
  • ...and 11 more figures