SURGIVID: Annotation-Efficient Surgical Video Object Discovery

TL;DR

The paper tackles the high annotation burden of pixel-level surgical video segmentation by introducing SURGIVID, an annotation-efficient pipeline that starts from self-supervised object discovery (DINO-based) to obtain pseudo masks via MaskCut, followed by a Mask2Former self-training refinement. It further leverages weak supervision from surgical phase labels to bias model attention toward instruments, improving tool localization by about 2%. On CaDIS Task II, the approach achieves performance comparable to fully-supervised methods using as little as 1% of labeled data, demonstrating substantial annotation savings (up to ~90%). The work combines patch-level self-supervision, graph-based object discovery, and weak phase supervision to enable scalable, accurate surgical object segmentation across large unlabeled video collections, with potential generalization to other procedures.

Abstract

Surgical scenes convey crucial information about the quality of surgery. Pixel-wise localization of tools and anatomical structures is the first task towards deeper surgical analysis for microscopic or endoscopic surgical views. This is typically done via fully-supervised methods which are annotation greedy and in several cases, demanding medical expertise. Considering the profusion of surgical videos obtained through standardized surgical workflows, we propose an annotation-efficient framework for the semantic segmentation of surgical scenes. We employ image-based self-supervised object discovery to identify the most salient tools and anatomical structures in surgical videos. These proposals are further refined within a minimally supervised fine-tuning step. Our unsupervised setup reinforced with only 36 annotation labels indicates comparable localization performance with fully-supervised segmentation models. Further, leveraging surgical phase labels as weak labels can better guide model attention towards surgical tools, leading to $\sim 2\%$ improvement in tool localization. Extensive ablation studies on the CaDIS dataset validate the effectiveness of our proposed solution in discovering relevant surgical objects with minimal or no supervision.

Figures (4)

• Figure 1: An illustration of DINO caron2021emerging attention maps for 12 different heads. a) Pretrained with ImageNet: Main anatomical structure and tools can be seen in the attention maps. b) Pretrained with ImageNet and fine-tuned on phase labels: attention maps are more focused on the surgical tools since tools include strong cues to detect the surgical phase of the surgery.
• Figure 2: Overview of suggested workflow for unsupervised surgical scene segmentation: a. A pretrained DINO (optionally fine-tuned on phase labels in a multi-task learning setting) is used to extract rich scene features. Features are then leveraged within MaskCut wang2023cut to generate initial course masks of salient objects within the scene. The produced masks are further refined within a self-training step using Mask2former cheng2022masked. b. The Mask2Former pre-trained with pseudo-labels can then be fine-tuned with $X\%$ annotated masks.
• Figure 3: Qualitative results of the unsupervised steps of suggested framework, namely the MaskCut and self-training. Self-training increases the segmentation quality and helps to find tools that are not detected with MaskCut.
• Figure 4: Qualitative results on Task II indicating the impact of annotation via feeding gradually increasing portions of labels to the self-trained model.