Text Promptable Surgical Instrument Segmentation with Vision-Language Models

TL;DR

This work tackles the variability and open-set nature of surgical instrument segmentation by reframing the task as text promptable segmentation using a CLIP-based vision-language backbone. The method combines a text-promptable mask decoder with attention and convolution prompting, a mixture of prompts that fuses multiple textual cues via a gating network, and a hard-area reinforcement module to sharpen boundary and detail understanding. Extensive experiments on EndoVis2017/2018 (and additional datasets) show state-of-the-art performance and strong cross-dataset generalization, with real-time inference capabilities and favorable computational efficiency. The approach demonstrates the potential of text-driven, promptable segmentation to enhance robustness and adaptability for robotic-assisted surgery, supported by open-source code and comprehensive ablations.

Abstract

In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

Figures (19)

• Figure 1: Left: input image and its ground truth. Top Right: conventional vision-based surgical instrument segmentation model with predefined categories. Bottom Right: ours leveraging vision-language model for text promptable surgical instrument segmentation.
• Figure 2: An overview of our method. Our method comprises four key modules: 1) image and text encoders derived from the pretrained vision-language model to obtain visual and textual features; 2) a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for predicting the score map from image features through text prompts; 3) a mixture of prompts mechanism that utilizes a visual-textual gating network to produce pixel-wise weights for merging different score maps; 4) a hard instrument area reinforcement module to reinforce image representation learning specifically on hard-predicted area.
• Figure 3: Ablation study for image and text encoder.
• Figure 4: Ablation study for text promptable mask decoder.
• Figure 5: Ablation study for mixture of prompts; cls - class, tem - template.