Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models

Sraavya Sambara, Sung Eun Kim, Xiaoman Zhang, Luyang Luo, Shreya Johri, Mohammed Baharoon, Du Hyun Ro, Pranav Rajpurkar

TL;DR

3DReasonKnee introduces the first large-scale 3D grounded reasoning dataset for medical imaging, enabling region-focused localization, stepwise diagnostic reasoning, and MOAKS-aligned severity judgments in knee MRI. By combining 3D volumetric data, voxel-level localization, clinician-authored reasoning traces, and structured outputs, it provides a rigorous testbed (ReasonKnee-Bench) to assess grounding and diagnostic accuracy in vision-language models. Extensive expert annotation (>450 hours) and a robust construction pipeline yield 7,970 DESS MRIs with 494k quintuples, designed to reflect real clinical workflows and MOAKS criteria. The work establishes a clear baseline for current VLM capabilities, highlights major grounding and reasoning gaps, and points to promising directions (e.g., RL, SFT, and region-aware training) to advance clinically aligned 3D multimodal AI.

Abstract

Current Vision-Language Models (VLMs) struggle to ground anatomical regions in 3D medical images and reason about them in a step-by-step manner, a key requirement of real-world diagnostic assessment. This ability is essential for aligning model outputs with the diagnostic workflows clinicians use in practice, enabling trustworthy clinician-AI collaboration. Existing 3D datasets provide localization labels, but none support this "grounded reasoning" ability. To address this gap, we introduce 3DReasonKnee, the first 3D grounded reasoning dataset for medical images, which provides 494k high-quality quintuples derived from 7,970 3D knee MRI volumes. Each quintuple includes: (1) the 3D MRI volume, (2) a diagnostic question targeting a specific anatomical region (3) a 3D bounding box localizing the relevant anatomical structures, (4) clinician-generated diagnostic reasoning steps that explicitly detail the 3D reasoning process, and (5) structured severity assessments for the relevant anatomical region. The creation and validation of 3DReasonKnee, involving over 450 hours of expert clinician time for manually segmenting MRIs and generating reasoning chains, ensures its superior quality and clinical relevance. We establish ReasonKnee-Bench to evaluate localization and diagnostic accuracy, providing insight into VLM ability to perform grounding and severity assessment across anatomical regions and diagnostic inquiries. We benchmark five state-of-the-art VLMs, providing baseline performance for ReasonKnee-Bench. By providing this unique resource of expert-annotated 3D reasoning pathways, 3DReasonKnee serves as a repository of orthopedic surgeons' diagnostic expertise and offers a vital testbed for advancing multimodal medical AI systems towards 3D, clinically aligned, localized decision-making capabilities. The dataset can be found in: https://huggingface.co/datasets/rajpurkarlab/3DReasonKnee

3DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Models

TL;DR

3DReasonKnee introduces the first large-scale 3D grounded reasoning dataset for medical imaging, enabling region-focused localization, stepwise diagnostic reasoning, and MOAKS-aligned severity judgments in knee MRI. By combining 3D volumetric data, voxel-level localization, clinician-authored reasoning traces, and structured outputs, it provides a rigorous testbed (ReasonKnee-Bench) to assess grounding and diagnostic accuracy in vision-language models. Extensive expert annotation (>450 hours) and a robust construction pipeline yield 7,970 DESS MRIs with 494k quintuples, designed to reflect real clinical workflows and MOAKS criteria. The work establishes a clear baseline for current VLM capabilities, highlights major grounding and reasoning gaps, and points to promising directions (e.g., RL, SFT, and region-aware training) to advance clinically aligned 3D multimodal AI.

Abstract

Current Vision-Language Models (VLMs) struggle to ground anatomical regions in 3D medical images and reason about them in a step-by-step manner, a key requirement of real-world diagnostic assessment. This ability is essential for aligning model outputs with the diagnostic workflows clinicians use in practice, enabling trustworthy clinician-AI collaboration. Existing 3D datasets provide localization labels, but none support this "grounded reasoning" ability. To address this gap, we introduce 3DReasonKnee, the first 3D grounded reasoning dataset for medical images, which provides 494k high-quality quintuples derived from 7,970 3D knee MRI volumes. Each quintuple includes: (1) the 3D MRI volume, (2) a diagnostic question targeting a specific anatomical region (3) a 3D bounding box localizing the relevant anatomical structures, (4) clinician-generated diagnostic reasoning steps that explicitly detail the 3D reasoning process, and (5) structured severity assessments for the relevant anatomical region. The creation and validation of 3DReasonKnee, involving over 450 hours of expert clinician time for manually segmenting MRIs and generating reasoning chains, ensures its superior quality and clinical relevance. We establish ReasonKnee-Bench to evaluate localization and diagnostic accuracy, providing insight into VLM ability to perform grounding and severity assessment across anatomical regions and diagnostic inquiries. We benchmark five state-of-the-art VLMs, providing baseline performance for ReasonKnee-Bench. By providing this unique resource of expert-annotated 3D reasoning pathways, 3DReasonKnee serves as a repository of orthopedic surgeons' diagnostic expertise and offers a vital testbed for advancing multimodal medical AI systems towards 3D, clinically aligned, localized decision-making capabilities. The dataset can be found in: https://huggingface.co/datasets/rajpurkarlab/3DReasonKnee
Paper Structure (15 sections, 4 figures, 3 tables)

This paper contains 15 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Existing Datasets and Benchmarks for Medical VLMs.
  4. Medical VLMs: Capabilities and Gaps.
  5. General Vision Language Models with Chain-of-Thought Reasoning.
  6. Dataset and Benchmark
  7. Problem Formulation: 3D Grounded Clinical Reasoning
  8. 3DReasonKnee Construction Pipeline
  9. ReasonKnee-Bench: Evaluating 3D Multimodal Grounded Clinical Reasoning
  10. Dataset Analysis
  11. Experiments and Baselines
  12. Baselines.
  13. Failure Modes.
  14. Discussion
  15. Acknowledgments

Figures (4)

  • Figure 1: Region-First Reasoning Workflow in Clinician Knee MRI Assessment. This schema illustrates the step-wise diagnostic process employed by clinicians, involving (Step 1) identification of lesion category and localization to specific anatomical subregions, (Step 2) assessment of severity and lesion type within the localized region, and (Step 3) characterization of specific features.
  • Figure 2: Example of 3DReasonKnee with Chain-of-Thought.
  • Figure 3: Overview of the 3DReasonKnee Dataset Construction Pipeline. MRIs (n=100) underwent manual segmentation to train an nn-Unet model isensee2021nnu, enabling automatic generation of 3D bounding boxes for all scans (n=7,970). Clinicians developed subregion-specific queries and structured CoT based on the MOAKS system.
  • Figure 4: Dataset distribution statistics. (a) Distribution of main MOAKS grades across different anatomical regions. The stacked bars indicate the frequency of different severity grades (1-3) for each anatomical location, with the remaining proportion representing grade 0. (b) Dice score of the nnU-Net model of all subregions.