Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework

TL;DR

This work reframes text-promptable surgical instrument segmentation as Robust SIS (R-SIS), where prompts are issued for all instrument classes without prior knowledge of presence. It introduces RoSIS, an encoder–decoder architecture that (i) predicts instrument existence for each prompt, (ii) segments only when an instrument is visually present, and (iii) employs iterative refinement with diverse prompts to improve accuracy under uncertainty. The authors define a robust evaluation protocol and benchmark on EndoVis2017 and EndoVis2018, showing RoSIS reduces false positives and achieves superior or competitive performance compared to vision-based and promptable baselines under realistic conditions. The work highlights the need for fair prompt-based evaluation in surgical scenes and provides a practical framework for robust, multi-modal instrument segmentation with potential to enhance interactive and automated robotic systems.

Abstract

Surgical instrument segmentation is an essential component of computer-assisted and robotic surgery systems. Vision-based segmentation models typically produce outputs limited to a predefined set of instrument categories, which restricts their applicability in interactive systems and robotic task automation. Promptable segmentation methods allow selective predictions based on textual prompts. However, they often rely on the assumption that the instruments present in the scene are already known, and prompts are generated accordingly, limiting their ability to generalize to unseen or dynamically emerging instruments. In practical surgical environments, where instrument existence information is not provided, this assumption does not hold consistently, resulting in false-positive segmentation. To address these limitations, we formulate a new task called Robust text-promptable Surgical Instrument Segmentation (R-SIS). Under this setting, prompts are issued for all candidate categories without access to instrument presence information. R-SIS requires distinguishing which prompts refer to visible instruments and generating masks only when such instruments are explicitly present in the scene. This setting reflects practical conditions where uncertainty in instrument presence is inherent. We evaluate existing segmentation methods under the R-SIS protocol using surgical video datasets and observe substantial false-positive predictions in the absence of ground-truth instruments. These findings demonstrate a mismatch between current evaluation protocols and real-world use cases, and support the need for benchmarks that explicitly account for prompt uncertainty and instrument absence.

Figures (5)

• Figure 1: Comparison of evaluation process of SIS method: (1) Vision-based segmentation model iglovikov2018ternausnetjin2019incorporatingzhao2020learninggonzalez2020isinet only uses the image and generates all classes of segmentation maps; (2) Previous promptable segmentation model zhou2023textyue2024surgicalsamyue2023part uses "oracle information" to identify categories present in the image and generates segmentation maps only for those categories. This means they are led to settings without robust conditions; (3) Our R-SIS uses prompts for all classes, generating segmentation maps for all categories under robust conditions.
• Figure 2: RoSIS model architecture with (a) overall structure, (b) Multi-Modal Fusion Block, and (c) Selective Gate Block.
• Figure 3: (a) Iterative refinement process of class $c$ (monopolar curved scissors in this image): (Iteration 1) Initial segmentation using a general prompt, (Iteration 2) refined segmentation with a location prompt referring to the first predicted map. Then, we combine maps by equation \ref{['eq:eq3']} for accurate final output. (b) After applying the iterative refinement process to all classes, the refined maps are combined to generate the final output.
• Figure 4: Visual comparison of iterative refinement strategies
• Figure 5: Qualitative results of our predicted maps, the ground truth masks, and other promptable segmentation methods from the EndoVis2018 validation set.