Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis

TL;DR

Med3D-R1 tackles the core challenges of 3D medical vision-language modeling by introducing RAM to stabilize cross-modal alignment, ARW to rebalance learning toward clinically meaningful abnormalities, and a consistency-rewarded RL stage to promote coherent, clinically grounded reasoning. The combination yields state-of-the-art MMVQA performance on CT-RATE and RAD-ChestCT, while also producing more interpretable rationales validated by radiologists and language models. The work demonstrates a practical path toward trustworthy, interpretable AI in complex 3D medical imaging, with potential to improve real-world diagnostic workflows. Limitations include data scale and domain-specific reward calibration; future work will expand anatomy, modalities, and clinician-driven rubric-based supervision to further enhance robustness and interpretability.

Abstract

Developing 3D vision-language models with robust clinical reasoning remains a challenge due to the inherent complexity of volumetric medical imaging, the tendency of models to overfit superficial report patterns, and the lack of interpretability-aware reward designs. In this paper, we propose Med3D-R1, a reinforcement learning framework with a two-stage training process: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). During SFT stage, we introduce a residual alignment mechanism to bridge the gap between high-dimensional 3D features and textual embeddings, and an abnormality re-weighting strategy to emphasize clinically informative tokens and reduce structural bias in reports. In RL stage, we redesign the consistency reward to explicitly promote coherent, step-by-step diagnostic reasoning. We evaluate our method on medical multiple-choice visual question answering using two 3D diagnostic benchmarks, CT-RATE and RAD-ChestCT, where our model attains state-of-the-art accuracies of 41.92\% on CT-RATE and 44.99\% on RAD-ChestCT. These results indicate improved abnormality diagnosis and clinical reasoning and outperform prior methods on both benchmarks. Overall, our approach holds promise for enhancing real-world diagnostic workflows by enabling more reliable and transparent 3D medical vision-language systems.

Figures (3)

• Figure 1: Density distribution of normal and abnormal sentences over normalized report positions. Normal sentences predominantly appear at the beginning of reports, whereas abnormal sentences are increasingly concentrated toward later positions, revealing a consistent positional bias in radiology report writing.
• Figure 2: Overview of the SFT framework. A residual alignment mechanism refines the mapping of 3D image features from the ViT to a fixed text anchor, thereby enhancing cross-modal alignment. An abnormality re-weighting strategy further emphasizes clinically significant findings in the generation loss. During this stage, only the adaptive weighted pooling module is trainable, while all other components remain frozen.
• Figure 3: Distribution of sentence- and token-level labels in CT-RATE reports. The outer ring of each donut illustrates the overall proportion of normal and abnormal sentences (left) and tokens (right), showing that the two categories occur in comparable quantities. The inner ring further decomposes each category into finding and impression components.