Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion

Meng Wei, Kun Yuan, Shi Li, Yue Zhou, Long Bai, Nassir Navab, Hongliang Ren, Hong Joo Lee, Tom Vercauteren, Nicolas Padoy

TL;DR

This work tackles surgical referring segmentation by leveraging instrument motion rather than static appearance cues. It introduces SurgRef, a motion-guided framework that grounds free-form language in temporal instrument dynamics, and Ref-IMotion, a multi-institutional dataset with dense spatiotemporal masks and motion-centric expressions. The approach combines a Swin Transformer visual backbone, a frozen RoBERTa language encoder, language-guided decoder queries, a key-frame attention mechanism, and inter-frame temporal fusion, achieving state-of-the-art performance and strong zero-shot generalization across procedures and toolsets. Results demonstrate that motion-centric supervision improves robustness to occlusion, unfamiliar terminology, and varying imaging modalities, advancing practical language-driven surgical scene understanding. This work provides a solid foundation for motion-aware, language-guided analysis in intelligent operating rooms and robotic-assisted surgery.

Abstract

Enabling intuitive, language-driven interaction with surgical scenes is a critical step toward intelligent operating rooms and autonomous surgical robotic assistance. However, the task of referring segmentation, localizing surgical instruments based on natural language descriptions, remains underexplored in surgical videos, with existing approaches struggling to generalize due to reliance on static visual cues and predefined instrument names. In this work, we introduce SurgRef, a novel motion-guided framework that grounds free-form language expressions in instrument motion, capturing how tools move and interact across time, rather than what they look like. This allows models to understand and segment instruments even under occlusion, ambiguity, or unfamiliar terminology. To train and evaluate SurgRef, we present Ref-IMotion, a diverse, multi-institutional video dataset with dense spatiotemporal masks and rich motion-centric expressions. SurgRef achieves state-of-the-art accuracy and generalization across surgical procedures, setting a new benchmark for robust, language-driven surgical video segmentation.

Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion

TL;DR

This work tackles surgical referring segmentation by leveraging instrument motion rather than static appearance cues. It introduces SurgRef, a motion-guided framework that grounds free-form language in temporal instrument dynamics, and Ref-IMotion, a multi-institutional dataset with dense spatiotemporal masks and motion-centric expressions. The approach combines a Swin Transformer visual backbone, a frozen RoBERTa language encoder, language-guided decoder queries, a key-frame attention mechanism, and inter-frame temporal fusion, achieving state-of-the-art performance and strong zero-shot generalization across procedures and toolsets. Results demonstrate that motion-centric supervision improves robustness to occlusion, unfamiliar terminology, and varying imaging modalities, advancing practical language-driven surgical scene understanding. This work provides a solid foundation for motion-aware, language-guided analysis in intelligent operating rooms and robotic-assisted surgery.

Abstract

Enabling intuitive, language-driven interaction with surgical scenes is a critical step toward intelligent operating rooms and autonomous surgical robotic assistance. However, the task of referring segmentation, localizing surgical instruments based on natural language descriptions, remains underexplored in surgical videos, with existing approaches struggling to generalize due to reliance on static visual cues and predefined instrument names. In this work, we introduce SurgRef, a novel motion-guided framework that grounds free-form language expressions in instrument motion, capturing how tools move and interact across time, rather than what they look like. This allows models to understand and segment instruments even under occlusion, ambiguity, or unfamiliar terminology. To train and evaluate SurgRef, we present Ref-IMotion, a diverse, multi-institutional video dataset with dense spatiotemporal masks and rich motion-centric expressions. SurgRef achieves state-of-the-art accuracy and generalization across surgical procedures, setting a new benchmark for robust, language-driven surgical video segmentation.
Paper Structure (25 sections, 5 equations, 3 figures, 4 tables)

This paper contains 25 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Segmentation.
  4. Surgical Video Segmentation.
  5. Surgical Video Referring Segmentation.
  6. Surgical Instrument Segmentation Dataset.
  7. Methodology
  8. Dataset Construction
  9. Classification Head.
  10. Mask Embedding Head.
  11. Key-frame Selection.
  12. Inter-frame Attention.
  13. Training Loss.
  14. Experiments
  15. Evaluation Metrics
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Data construction process of the EndoVis-IM17, EndoVis-IM18, GraSP-IM, and CholecSeg8k-IM.
  • Figure 2: Baseline model for SurgRef, which integrates a Swin Transformer backbone for spatial feature extraction, a frozen RoBERTa encoder for language grounding, a language-guided transformer decoder for cross-modal reasoning, and a key-frame attention module for selecting semantically aligned frames.
  • Figure 3: Visualization examples of our SurgRef on EndoVis-IM18 and GraSP-IM dataset.