Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation

Qingqiu Li, Zihang Cui, Seongsu Bae, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, Junjun He, Shujun Wang

TL;DR

The paper tackles the lack of region-level understanding and interpretable multi-step reasoning in medical large multimodal models for chest X-ray interpretation. It introduces Anatomical Ontology-Guided Reasoning (AOR), a three-stage framework that centers on anatomical regions and leverages a specialized AOR-Instruction dataset to train models for medical VQA and region-based report generation. The approach combines an image encoder, a region encoder, and an LLM with region-aware projections, enabling multimodal, multi-step reasoning and explicit region grounding. Empirical results show that AOR achieves higher accuracy and more faithful, region-specific reports than prior methods, with strong generalization and grounded qualitative behavior, suggesting substantial potential for clinical decision support. The work also provides a scalable data-and-methodology blueprint for region-centric reasoning in multimodal medical imaging models.

Abstract

Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements in Large Multimodal Models (LMMs) have enabled automated CXR interpretation, enhancing diagnostic accuracy and efficiency. However, despite their strong visual understanding, current Medical LMMs (MLMMs) still face two major challenges: (1) Insufficient region-level understanding and interaction, and (2) Limited accuracy and interpretability due to single-step reasoning. In this paper, we empower MLMMs with anatomy-centric reasoning capabilities to enhance their interactivity and explainability. Specifically, we first propose an Anatomical Ontology-Guided Reasoning (AOR) framework, which centers on cross-modal region-level information to facilitate multi-step reasoning. Next, under the guidance of expert physicians, we develop AOR-Instruction, a large instruction dataset for MLMMs training. Our experiments demonstrate AOR's superior performance in both VQA and report generation tasks.

AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation

TL;DR

The paper tackles the lack of region-level understanding and interpretable multi-step reasoning in medical large multimodal models for chest X-ray interpretation. It introduces Anatomical Ontology-Guided Reasoning (AOR), a three-stage framework that centers on anatomical regions and leverages a specialized AOR-Instruction dataset to train models for medical VQA and region-based report generation. The approach combines an image encoder, a region encoder, and an LLM with region-aware projections, enabling multimodal, multi-step reasoning and explicit region grounding. Empirical results show that AOR achieves higher accuracy and more faithful, region-specific reports than prior methods, with strong generalization and grounded qualitative behavior, suggesting substantial potential for clinical decision support. The work also provides a scalable data-and-methodology blueprint for region-centric reasoning in multimodal medical imaging models.

Abstract

Chest X-rays (CXRs) are the most frequently performed imaging examinations in clinical settings. Recent advancements in Large Multimodal Models (LMMs) have enabled automated CXR interpretation, enhancing diagnostic accuracy and efficiency. However, despite their strong visual understanding, current Medical LMMs (MLMMs) still face two major challenges: (1) Insufficient region-level understanding and interaction, and (2) Limited accuracy and interpretability due to single-step reasoning. In this paper, we empower MLMMs with anatomy-centric reasoning capabilities to enhance their interactivity and explainability. Specifically, we first propose an Anatomical Ontology-Guided Reasoning (AOR) framework, which centers on cross-modal region-level information to facilitate multi-step reasoning. Next, under the guidance of expert physicians, we develop AOR-Instruction, a large instruction dataset for MLMMs training. Our experiments demonstrate AOR's superior performance in both VQA and report generation tasks.
Paper Structure (17 sections, 6 equations, 6 figures, 4 tables)

This paper contains 17 sections, 6 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Model Overview
  5. Model Development
  6. Instruction Data
  7. AOR-VQA
  8. Anatomical Ontologies Design
  9. CoT construction
  10. Sample Expansion
  11. AOR-RG
  12. Experiments
  13. Experiment Settings
  14. Quantitative Comparison
  15. Ablation Studies and Discussions
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Previous image-level MLMMs (shown in light red) make incorrect predictions or fail to predict due to (1) insufficient region-level perception and (2) reliance on single-step reasoning. In contrast, our AOR model (shown in light green) delivers explainable and accurate answers by (1) emphasizing region-level understanding and (2) employing multi-step reasoning.
  • Figure 2: (a) Overview of AOR framework, which flexibly accommodates both textual and optional visual prompts as input, centered on region-level information to enable multimodal multi-step reasoning and (b) Three-stage training procedure for AOR.
  • Figure 2: Comparison of methods on MIMIC-CXR dataset.
  • Figure 3: Overview of AOR-Instruction, which consists of two sub-datasets: (a) The construction of AOR-VQA: Anatomical ontologies design $\rightarrow$ CoT construction $\rightarrow$ Sample expansion and (b) The construction of AOR-RG: Strict alignment between anatomical region and report sentence.
  • Figure 4: Impact of anatomical region shifts on model predictions.
  • ...and 1 more figures