Slot-BERT: Self-supervised Object Discovery in Surgical Video

TL;DR

Slot-BERT tackles unsupervised object discovery in long surgical videos by learning object-centric slot representations and fusing them over time with a bidirectional Temporal Slot Transformer (TST). It couples slot attention-based encoding with a masked transformer for robust long-range temporal reasoning and introduces a slot-contrastive loss to enforce slot orthogonality, reducing redundancy. The approach achieves state-of-the-art unsupervised segmentation and strong zero-shot transfer across MICCAI SurgToolLoc, Cholec80, EndoVis, and Thoracic datasets, while remaining computationally efficient for real-world deployment. By avoiding heavy cues like optical flow and enabling scalable long-sequence processing, Slot-BERT offers a practical framework for explainable, self-supervised object-centric learning in medical video analysis, with promising avenues for domain adaptation and higher-resolution boundary refinement.

Abstract

Object-centric slot attention is a powerful framework for unsupervised learning of structured and explainable representations that can support reasoning about objects and actions, including in surgical videos. While conventional object-centric methods for videos leverage recurrent processing to achieve efficiency, they often struggle with maintaining long-range temporal coherence required for long videos in surgical applications. On the other hand, fully parallel processing of entire videos enhances temporal consistency but introduces significant computational overhead, making it impractical for implementation on hardware in medical facilities. We present Slot-BERT, a bidirectional long-range model that learns object-centric representations in a latent space while ensuring robust temporal coherence. Slot-BERT scales object discovery seamlessly to long videos of unconstrained lengths. A novel slot contrastive loss further reduces redundancy and improves the representation disentanglement by enhancing slot orthogonality. We evaluate Slot-BERT on real-world surgical video datasets from abdominal, cholecystectomy, and thoracic procedures. Our method surpasses state-of-the-art object-centric approaches under unsupervised training achieving superior performance across diverse domains. We also demonstrate efficient zero-shot domain adaptation to data from diverse surgical specialties and databases.

Figures (13)

• Figure 1: Overview of our object-centric representation learning framework. Video sequences are encoded into features, processed with a recurrent slot attention mechanism, and refined using a Temporal Slot Transformer (TST). The final slot representations are decoded to reconstruct the input features, with training optimized to minimize reconstruction loss and slot contrastive loss.
• Figure 2: Qualitative results of unsupervised segmentation using STEVE, SAVi and our method on the Cholec dataset, which can be evaluated both tissue and instrument ground truth (GT). Slot-BERT demonstrates a stronger ability to delineate instruments from tissue and to separate tissues with different textures.
• Figure 3: Qualitative results of zero-shot experiments on unseen datasets using our method compared to STEVE and SAVi. Slot-BERT demonstrates superior adaptability, successfully segmenting unseen surgical instruments, while alternative methods exhibit degraded performance. Despite limitations in precise boundary detection due to patch-based processing, our method achieves high object localization coverage.
• Figure 4: Impact of sequence length on performance. The plot illustrates mBO-F and mBO-V metrics across different sequence lengths for various methods. Our method demonstrates stronger temporal consistency, with minimal degradation in video-level accuracy (mBO-V) as sequence length increases, outperforming SOTA approaches across all evaluated settings.
• Figure 5: Slot masks and the corresponding cosine similarity matrices for Slot-BERT and ablated Slot-BERT variants without the TST module (left) or contrastive loss (right). All models are trained with 7 slots, and each segmentation mask is annotated with its corresponding slot index (0–6). See text for details.