WeakSurg: Weakly supervised surgical instrument segmentation using temporal equivariance and semantic continuity

TL;DR

This work tackles weakly supervised surgical instrument segmentation using only instrument presence labels by leveraging temporal dynamics. It introduces WeakSurg, a two-stage WSSS framework built on a Multi-Class Token Transformer and augmented with a Temporal Equivariance Constraint, a Class-aware Semantic Continuity Constraint, and temporal-enhanced pseudo mask generation. Across Cholec80 and RLLS, WeakSurg delivers consistent improvements in semantic and instance segmentation metrics over prior methods, demonstrating the value of temporal priors in reducing annotation costs. The approach enables robust instrument localization and segmentation with reduced manual labeling, supporting more scalable deployment in robotic surgery.

Abstract

For robotic surgical videos, instrument presence annotations are typically recorded with video streams, which offering the potential to reduce the manually annotated costs for segmentation. However, weakly supervised surgical instrument segmentation with only instrument presence labels has been rarely explored in surgical domain due to the highly under-constrained challenges. Temporal properties can enhance representation learning by capturing sequential dependencies and patterns over time even in incomplete supervision situations. From this, we take the inherent temporal attributes of surgical video into account and extend a two-stage weakly supervised segmentation paradigm from different perspectives. Firstly, we make temporal equivariance constraint to enhance pixel-wise temporal consistency between adjacent features. Secondly, we constrain class-aware semantic continuity between global and local regions across temporal dimension. Finally, we generate temporal-enhanced pseudo masks from consecutive frames to suppress irrelevant regions. Extensive experiments are validated on two surgical video datasets, including one cholecystectomy surgery benchmark and one real robotic left lateral segment liver surgery dataset. We annotate instance-wise instrument labels with fixed time-steps which are double checked by a clinician with 3-years experience to evaluate segmentation results. Experimental results demonstrate the promising performances of our method, which consistently achieves comparable or favorable results with previous state-of-the-art approaches.

Figures (6)

• Figure 1: Motivation of weakly-supervised instrument segmentation which only rely on instrument presence annotations. The recorded presence information can be obtained automatically through OCR methods with low cost. This paradigm can alleviate manual pixel-wise annotations for segmentation methods.
• Figure 2: Schematic illustration of an overview pipeline of our method. The main flow is based on a two-stage WSSS method, MCT, as presented in purple chart. Two temporality-based constraints are inserted into baseline in train process.
• Figure 3: The flowchart of our temporal-enhanced pseudo masks generation.
• Figure 4: Qualitative comparisons of segmentation networks for both Cholec80 and RLLS. In comparison, our proposed network achieves the best prediction at most situations.
• Figure 5: Qualitative comparison for frame-wise and temporal-enhanced CAMs. The red box represent the irrelevant regions which need to suppress for current category.