VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models

TL;DR

IR-SIS introduces a VLM-guided iterative refinement framework for language-based surgical image segmentation. It combines a SAM3-based segmenter fine-tuned on multi-level language annotations with a Vision-Language instrument detector, a quality evaluator, and an agentic refinement workflow that can incorporate clinician feedback. The method achieves state-of-the-art performance on EndoVis2017/2018 in-domain data and demonstrates strong generalization to the Kvasir-Instrument out-of-distribution dataset, illustrating the practical utility of flexible natural-language prompts and adaptive refinement in evolving surgical environments. This work advances interactive surgical segmentation by enabling flexible queries, autonomous quality-driven refinement, and clinician-guided correction when necessary.

Abstract

Surgical image segmentation is essential for robot-assisted surgery and intraoperative guidance. However, existing methods are constrained to predefined categories, produce one-shot predictions without adaptive refinement, and lack mechanisms for clinician interaction. We propose IR-SIS, an iterative refinement system for surgical image segmentation that accepts natural language descriptions. IR-SIS leverages a fine-tuned SAM3 for initial segmentation, employs a Vision-Language Model to detect instruments and assess segmentation quality, and applies an agentic workflow that adaptively selects refinement strategies. The system supports clinician-in-the-loop interaction through natural language feedback. We also construct a multi-granularity language-annotated dataset from EndoVis2017 and EndoVis2018 benchmarks. Experiments demonstrate state-of-the-art performance on both in-domain and out-of-distribution data, with clinician interaction providing additional improvements. Our work establishes the first language-based surgical segmentation framework with adaptive self-refinement capabilities.

Figures (6)

• Figure 1: Existing language-based surgical segmentation methods face two key limitations: a) They are restricted to predefined instrument categories (e.g., TP-SIS), producing one-shot predictions without adaptive refinement or clinician interaction. (b) Our approach IR-SIS enables flexible natural language descriptions, employs an agentic workflow for adaptive self-refinement, and supports clinician-in-the-loop interaction for quality improvement.
• Figure 2: IR-SIS Pipeline Overview. We first fine-tune SAM3 on our dataset to perform initial segmentation on input surgical images. Subsequently, a VLM is employed to detect all surgical instruments. Segmentation quality is then evaluated based on mask coverage and mask-box overlap ratio: if the quality meets the criteria, only morphological post-processing is applied; otherwise, each instrument is re-segmented using box prompts followed by iterative refinement. Additionally, clinicians can provide feedback during the iterative process to guide the refinement direction, ultimately yielding refined segmentation results.
• Figure 3: Detailed workflow of the SAM3 Refinement Agent. The agent adaptively selects between trust-initial and multi-instrument strategies based on mask-box overlap quality.
• Figure 4: Text prompt segmentation results on Endovis2017 and Endovis2018 Datasets
• Figure 5: Text prompt segmentation results on Kvasir-instrument dataset