MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

TL;DR

MedVL-SAM2 introduces a unified 3D medical vision–language model that combines a volumetric vision encoder with SAM2-based segmentation to jointly tackle report generation, VQA, and semantic/referring/interactive segmentation over volumetric CT data. The problem is formulated as $(\mathcal{T}_{out},\mathcal{M}_{out}) = \mathcal{F}(\Theta;\mathcal{I}_{in},\mathcal{T}_{in},\mathcal{P}_{in})$, where $\Theta=\{\theta_{vlb}, \theta_{seg}\}$ and the system fuses 3D visual features with language via a LoRA-enhanced LLM; segmentation is produced through a SAM2 decoder conditioned on the [SEG] token. A 3D CLIP-based vision encoder with an MLP-Mixer projection reduces $n$ from 2048 to 512 tokens ($\hat{n}=512$, $\hat{d}=2048$) for efficient cross-modal alignment, and cross-slice consistency is maintained with memory attention in SAM2. Training proceeds in three stages to align 3D visual representations, language understanding, and spatial grounding, followed by joint optimization with a text-loss $\mathcal{L}_{text}$ and a mask loss $\mathcal{L}_{joint}=\lambda_{text}\mathcal{L}_{text}+\lambda_{mask}(\mathcal{L}_{CE}+\lambda_{Dice}\mathcal{L}_{Dice})$. On CT-RATE and M3D-Seg benchmarks, MedVL-SAM2 achieves state-of-the-art performance across report generation, VQA, and multiple segmentation tasks, with reliable 3D grounding and effective interactive segmentation, indicating strong potential for clinical deployment.

Abstract

Recent progress in medical vision-language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report generation and visual question answering (VQA). However, achieving fine-grained visual grounding and volumetric spatial reasoning in 3D medical VLMs remains challenging, particularly when aiming to unify these capabilities within a single, generalizable framework. To address this challenge, we proposed MedVL-SAM2, a unified 3D medical multimodal model that concurrently supports report generation, VQA, and multi-paradigm segmentation, including semantic, referring, and interactive segmentation. MedVL-SAM2 integrates image-level reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging, and incorporates a SAM2-based volumetric segmentation module to enable precise multi-granular spatial reasoning. The model is trained in a multi-stage pipeline: it is first pre-trained on a large-scale corpus of 3D CT image-text pairs to align volumetric visual features with radiology-language embeddings. It is then jointly optimized with both language-understanding and segmentation objectives using a comprehensive 3D CT segmentation dataset. This joint training enables flexible interaction via language, point, or box prompts, thereby unifying high-level visual reasoning with spatially precise localization. Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D segmentation tasks. Extensive analyses further show that the model provides reliable 3D visual grounding, controllable interactive segmentation, and robust cross-modal reasoning, demonstrating that high-level semantic reasoning and precise 3D localization can be jointly achieved within a unified 3D medical VLM.

Figures (10)

• Figure 1: Comparison of the proposed model to traditional text-response-only medical vision-language models (VLMs) and mask-response-only medical vision foundation models (VFMs).
• Figure 2: Overview of the proposed architecture, which integrates a LLaVA-style VLM with a SAM2-based segmentation module. The VLM processes 3D volumes and generates text autoregressively. When a [SEG] token is produced, its hidden state is extracted and fed into SAM2’s prompt encoder, where it is fused with optional visual prompts (points or boxes) to generate the final segmentation mask.
• Figure 3: Comparison between the proposed methods to CT-CHAT on VQA tasks, including Long Answer, Short Answer, and Multiple choice subsets. Critical clinical information were highlighted in red. For CT-CHAT, the input question additionally contained a special token ([short_answer], [long_answer], [multiple_choice]) to indicate the specific subset to evaluate.
• Figure 4: Comparison of the proposed method with M3D on referring and semantic segmentation. Interactive segmentation results using bounding-box prompts are also included for comparison (CTOrg dataset shown). Liver is shown in magenta, bladder in green, lung in brown, kidney in purple, bone in red, and brain in blue.
• Figure 5: Prompt templates used for report generation tasks.