PRS-Med: Position Reasoning Segmentation with Vision-Language Model in Medical Imaging

TL;DR

PRS-Med tackles position-aware segmentation in medical imaging by fusing a lightweight vision backbone with a vision-language model and a segmentation decoder. It introduces the MMRS dataset pipeline to generate spatially grounded, natural-language prompts and QA pairs across six imaging modalities. Empirically, PRS-Med outperforms state-of-the-art segmentation and reasoning baselines, with ablations validating the contributions of the cross-attention fusion, backbone choices, and LoRA training. The work enables intuitive doctor-system interaction and provides open-source resources to catalyze future research in spatially-aware multimodal medical AI.

Abstract

Recent advancements in prompt-based medical image segmentation have enabled clinicians to identify tumors using simple input like bounding boxes or text prompts. However, existing methods face challenges when doctors need to interact through natural language or when position reasoning is required - understanding spatial relationships between anatomical structures and pathologies. We present PRS-Med, a framework that integrates vision-language models with segmentation capabilities to generate both accurate segmentation masks and corresponding spatial reasoning outputs. Additionally, we introduce the MMRS dataset (Multimodal Medical in Positional Reasoning Segmentation), which provides diverse, spatially-grounded question-answer pairs to address the lack of position reasoning data in medical imaging. PRS-Med demonstrates superior performance across six imaging modalities (CT, MRI, X-ray, ultrasound, endoscopy, RGB), significantly outperforming state-of-the-art methods in both segmentation accuracy and position reasoning. Our approach enables intuitive doctor-system interaction through natural language, facilitating more efficient diagnoses. Our dataset pipeline, model, and codebase will be released to foster further research in spatially-aware multimodal reasoning for medical applications.

Figures (9)

• Figure 1: Example interaction between the doctor and proposed assistant model. Given a natural language prompt about a tumour and its location, along with an input image, the assistant returns a segmentation result and a spatially grounded textual description.
• Figure 2: Position reasoning dataset question-answer pair creation consists of two stages: (1) dictionary creation, which stores question–answer templates; and (2) data generation, which produces question–answer pairs using positional information extracted from segmentation masks.
• Figure 3: The architecture of PRS-Med comprises three primary components: (1) the Tiny Vision Backbone, (2) the Prompt Mask Decoder, and (3) the Multimodal-LLM. The framework accepts two input modalities: an image and a text-based prompt (e.g., a question). The image is processed through a vision encoder, while the prompt is embedded via a LoRA-adapted Multimodal-LLM. The fused representations are used to produce two outputs: a segmentation mask for the tumor regions, and a textual description specifying the tumor's location.
• Figure 4: Qualitative comparison between PRS-Med in the segmentation results with previous methods.
• Figure 5: Overall design of the Prompt Mask Decoder block, including two modules are fusion module to fuse the medical image representation and the conditioning embedding from the Multimodal-LLM; the other is the mask prediction to predict the segment mask.