Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness

Idris Hamoud, Alexandros Karargyris, Aidean Sharghi, Omid Mohareri, Nicolas Padoy

TL;DR

This work tackles annotation scarcity in operating room scene understanding by introducing a depth-driven 3D self-supervised pretext task that predicts the Euclidean distance between depth-derived superpixel clusters. An encoder-decoder network processes ToF depth maps, with a novel Superpixel Sampling module enforcing alignment of cluster features across views, enabling effective pretraining for semantic segmentation and activity classification. The method is evaluated on two OR depth datasets and compared against RotNet and CPC v2, showing notable gains in low-data regimes and robust performance across architectures. The approach reduces annotation requirements for OR context awareness and holds practical potential for data-efficient, privacy-preserving surgical AI systems.

Abstract

Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow. In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting the relative 3D distance of image patches by exploiting the depth maps. Learning 3D spatial context generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multi-view data captured from clinical scenarios. We demonstrate a noteworthy improvement of performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest.

Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness

TL;DR

This work tackles annotation scarcity in operating room scene understanding by introducing a depth-driven 3D self-supervised pretext task that predicts the Euclidean distance between depth-derived superpixel clusters. An encoder-decoder network processes ToF depth maps, with a novel Superpixel Sampling module enforcing alignment of cluster features across views, enabling effective pretraining for semantic segmentation and activity classification. The method is evaluated on two OR depth datasets and compared against RotNet and CPC v2, showing notable gains in low-data regimes and robust performance across architectures. The approach reduces annotation requirements for OR context awareness and holds practical potential for data-efficient, privacy-preserving surgical AI systems.

Abstract

Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow. In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting the relative 3D distance of image patches by exploiting the depth maps. Learning 3D spatial context generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multi-view data captured from clinical scenarios. We demonstrate a noteworthy improvement of performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest.
Paper Structure (23 sections, 6 equations, 5 figures)

This paper contains 23 sections, 6 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Operating Room Context Awareness
  4. Self-supervised learning
  5. Methodology
  6. Proposed Pretext Task
  7. Self-supervised labeling strategy
  8. Encoder-Decoder architecture
  9. Feature Extraction:
  10. Architecture:
  11. Superpixel Sampling module (SPS):
  12. Semi-supervised Learning: Semantic Segmentation & Activity Classification
  13. Methodology:
  14. Evaluation Metrics:
  15. Experiments and Results
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Pretext task annotation generation process using SLIC SLIC2012 superpixel segmentation.
  • Figure 2: Framework used for self-supervised learning; numbers 4 and 5 on feature vectors refer to Figure \ref{['fig1:imgs']}.
  • Figure 3: Overview of the semantic segmentation and activity classification datasets Li2020AR3Sharghi2020.
  • Figure 4: t-SNE JMLR:v9:vandermaaten08a visualization of the superpixel features learnt from the pretext task.
  • Figure 5: Median mIoU and mAP with Interquartile Range (IQR) as a function of training available labels as described section 4.3. Our method outperforms the baseline method without pretraining and is on par with other self-supervised methods.