Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking Like a Radiologist: A Dataset for Anatomy-Guided Interleaved Vision Language Reasoning in Chest X-ray Interpretation

Yichen Zhao, Zelin Peng, Piao Yang, Xiaokang Yang, Wei Shen

Abstract

Radiological diagnosis is a perceptual process in which careful visual inspection and language reasoning are repeatedly interleaved. Most medical large vision language models (LVLMs) perform visual inspection only once and then rely on text-only chain-of-thought (CoT) reasoning, which operates purely in the linguistic space and is prone to hallucination. Recent methods attempt to mitigate this issue by introducing visually related coordinates, such as bounding boxes. However, these remain a pseudo-visual solution: coordinates are still text and fail to preserve rich visual details like texture and density. Motivated by the interleaved nature of radiological diagnosis, we introduce MMRad-IVL-22K, the first large-scale dataset designed for natively interleaved visual language reasoning in chest X-ray interpretation. MMRad-IVL-22K reflects a repeated cycle of reasoning and visual inspection workflow of radiologists, in which visual rationales complement textual descriptions and ground each step of the reasoning process. MMRad-IVL-22K comprises 21,994 diagnostic traces, enabling systematic scanning across 35 anatomical regions. Experimental results on advanced closed-source LVLMs demonstrate that report generation guided by multimodal CoT significantly outperforms that guided by text-only CoT in clinical accuracy and report quality (e.g., 6\% increase in the RadGraph metric), confirming that high-fidelity interleaved vision language evidence is a non-substitutable component of reliable medical AI. Furthermore, benchmarking across seven state-of-the-art open-source LVLMs demonstrates that models fine-tuned on MMRad-IVL-22K achieve superior reasoning consistency and report quality compared with both general-purpose and medical-specific LVLMs. The project page is available at https://github.com/qiuzyc/thinking_like_a_radiologist.

Thinking Like a Radiologist: A Dataset for Anatomy-Guided Interleaved Vision Language Reasoning in Chest X-ray Interpretation

Abstract

Radiological diagnosis is a perceptual process in which careful visual inspection and language reasoning are repeatedly interleaved. Most medical large vision language models (LVLMs) perform visual inspection only once and then rely on text-only chain-of-thought (CoT) reasoning, which operates purely in the linguistic space and is prone to hallucination. Recent methods attempt to mitigate this issue by introducing visually related coordinates, such as bounding boxes. However, these remain a pseudo-visual solution: coordinates are still text and fail to preserve rich visual details like texture and density. Motivated by the interleaved nature of radiological diagnosis, we introduce MMRad-IVL-22K, the first large-scale dataset designed for natively interleaved visual language reasoning in chest X-ray interpretation. MMRad-IVL-22K reflects a repeated cycle of reasoning and visual inspection workflow of radiologists, in which visual rationales complement textual descriptions and ground each step of the reasoning process. MMRad-IVL-22K comprises 21,994 diagnostic traces, enabling systematic scanning across 35 anatomical regions. Experimental results on advanced closed-source LVLMs demonstrate that report generation guided by multimodal CoT significantly outperforms that guided by text-only CoT in clinical accuracy and report quality (e.g., 6\% increase in the RadGraph metric), confirming that high-fidelity interleaved vision language evidence is a non-substitutable component of reliable medical AI. Furthermore, benchmarking across seven state-of-the-art open-source LVLMs demonstrates that models fine-tuned on MMRad-IVL-22K achieve superior reasoning consistency and report quality compared with both general-purpose and medical-specific LVLMs. The project page is available at https://github.com/qiuzyc/thinking_like_a_radiologist.
Paper Structure (10 sections, 3 figures, 3 tables)

This paper contains 10 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. MMRad-IVL-22K Dataset
  3. Data Construction
  4. Data Statistics
  5. Data Comparison
  6. Experiments
  7. Value Analysis of Multimodal CoT
  8. Benchmarking LVLMs on MMRad-IVL-22K
  9. Case Study
  10. Conclusion

Figures (3)

  • Figure 1: Pipeline of MMRad-IVL-22K. (1) Initial interleaved reasoning traces and region-text correspondences are generated and summarized based on GEMeX-ThinkVG via DeepSeek-v3. (2) Multi-stage verification is conducted through DeepSeek-v3 self-reflection, Qwen2.5-72B cross-auditing, and domain expert adjudication to ensure clinical fidelity. (3) Refined rationales and bounding boxes are aggregated into major anatomical categories. (4) Localized reasoning images are cropped and assembled with textual traces and medical reports to form the final interleaved MMRad-IVL-22K dataset.
  • Figure 2: Statistical overview of MMRad-IVL-22K. (a-b) Distribution of reasoning steps and rationale lengths. (c) Fine-grained anatomical structures. (d-e) Clinical characteristics extracted across the dataset.
  • Figure 3: Qualitative analysis of the interleaved reasoning process and final report generated by Anole-RadCoT. This systematic trace demonstrates high clinical fidelity through precise visual grounding. Green shading and blue shading represent consistent descriptions for the mediastinal and lung regions; red shading indicates partial consistency with similar meanings.