Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Medical Image Spatial Grounding with Semantic Sampling

Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura, Mingrui Yang, Xiaojuan Li, Vipin Chaudhary

Abstract

Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce \textbf{MIS-Ground}, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of \textbf{M}edical \textbf{I}mage \textbf{S}patial \textbf{Ground}ing. We release MIS-Ground to the public at \href{https://anonymous.4open.science/r/mis-ground}{\texttt{anonymous.4open.science/r/mis-ground}}. In addition, we present \textbf{MIS-SemSam}, a low-cost, inference-time, and model-agnostic optimization of VLMs that improve their spatial grounding ability with the use of \textbf{Sem}antic \textbf{Sam}pling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06\%.

Medical Image Spatial Grounding with Semantic Sampling

Abstract

Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce \textbf{MIS-Ground}, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of \textbf{M}edical \textbf{I}mage \textbf{S}patial \textbf{Ground}ing. We release MIS-Ground to the public at \href{https://anonymous.4open.science/r/mis-ground}{\texttt{anonymous.4open.science/r/mis-ground}}. In addition, we present \textbf{MIS-SemSam}, a low-cost, inference-time, and model-agnostic optimization of VLMs that improve their spatial grounding ability with the use of \textbf{Sem}antic \textbf{Sam}pling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06\%.
Paper Structure (10 sections, 4 equations, 2 figures)

This paper contains 10 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Dataset Preparation
  4. Benchmark Curation
  5. VLM Evaluation
  6. Semantic Sampling
  7. Results
  8. Conclusions

Figures (2)

  • Figure 1: Sample question-answer pairs from MIS-Ground. For each 2D or 3D vision input (top), multiple questions are generated (bottom). Video VLMs display the ability to discern anatomical structure relationships across slices (RQ1); they prefer anatomical direction terms like anterior (RQ2); and they are capable of abstract spatial reasoning (AB2). Note: the CT image is shown in axial RAS standard viewing orientation, whereas the MRI is shown in sagittal RAS storage mode.
  • Figure 2: Overall accuracy on the MIS-Ground benchmark, by model family and size. Note that Med3DVLM (0.0%) and M3D (15.9%) were tested but omitted from this graph. MIS-Semsam family are derived from the Qwen3-VL family.