Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Learning for Endoscopic Video Analysis

Roy Hirsch, Mathilde Caron, Regev Cohen, Amir Livne, Ron Shapiro, Tomer Golany, Roman Goldenberg, Daniel Freedman, Ehud Rivlin

TL;DR

This work studies the use of a leading SSL framework, namely Masked Siamese Networks (MSNs), for endoscopic video analysis such as colonoscopy and laparoscopy, and provides evidence that SSL can dramatically reduce the need of annotated data in endoscopy.

Abstract

Self-supervised learning (SSL) has led to important breakthroughs in computer vision by allowing learning from large amounts of unlabeled data. As such, it might have a pivotal role to play in biomedicine where annotating data requires a highly specialized expertise. Yet, there are many healthcare domains for which SSL has not been extensively explored. One such domain is endoscopy, minimally invasive procedures which are commonly used to detect and treat infections, chronic inflammatory diseases or cancer. In this work, we study the use of a leading SSL framework, namely Masked Siamese Networks (MSNs), for endoscopic video analysis such as colonoscopy and laparoscopy. To fully exploit the power of SSL, we create sizable unlabeled endoscopic video datasets for training MSNs. These strong image representations serve as a foundation for secondary training with limited annotated datasets, resulting in state-of-the-art performance in endoscopic benchmarks like surgical phase recognition during laparoscopy and colonoscopic polyp characterization. Additionally, we achieve a 50% reduction in annotated data size without sacrificing performance. Thus, our work provides evidence that SSL can dramatically reduce the need of annotated data in endoscopy.

Self-Supervised Learning for Endoscopic Video Analysis

TL;DR

This work studies the use of a leading SSL framework, namely Masked Siamese Networks (MSNs), for endoscopic video analysis such as colonoscopy and laparoscopy, and provides evidence that SSL can dramatically reduce the need of annotated data in endoscopy.

Abstract

Self-supervised learning (SSL) has led to important breakthroughs in computer vision by allowing learning from large amounts of unlabeled data. As such, it might have a pivotal role to play in biomedicine where annotating data requires a highly specialized expertise. Yet, there are many healthcare domains for which SSL has not been extensively explored. One such domain is endoscopy, minimally invasive procedures which are commonly used to detect and treat infections, chronic inflammatory diseases or cancer. In this work, we study the use of a leading SSL framework, namely Masked Siamese Networks (MSNs), for endoscopic video analysis such as colonoscopy and laparoscopy. To fully exploit the power of SSL, we create sizable unlabeled endoscopic video datasets for training MSNs. These strong image representations serve as a foundation for secondary training with limited annotated datasets, resulting in state-of-the-art performance in endoscopic benchmarks like surgical phase recognition during laparoscopy and colonoscopic polyp characterization. Additionally, we achieve a 50% reduction in annotated data size without sacrificing performance. Thus, our work provides evidence that SSL can dramatically reduce the need of annotated data in endoscopy.
Paper Structure (21 sections, 5 figures, 5 tables)

This paper contains 21 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Cholecystectomy Phase Recognition
  4. Optical Polyp Characterization
  5. Self-Supervised Learning for Endoscopy
  6. Masked Siamese Networks
  7. Private Datasets
  8. Laparoscopy.
  9. Colonoscopy.
  10. Experiments
  11. Downstream Task Evaluation Protocols.
  12. Implementation Details.
  13. Results and Discussion
  14. Scaling laws of SSL.
  15. Low-Shot Regime.
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Schematic of Masked Siamese Networks.
  • Figure 2: Data Samples. Top: Laparoscopy. Bottom: Colonoscopy.
  • Figure 3: Low-shot evaluation comparing MSN to fully supervised baselines.
  • Figure 4: The phases of the public laparoscopic dataset Cholec80 twinanda2016endonet. We report the mean and standard deviation of each phase's duration in seconds.
  • Figure 5: A description of the procedures that comprise the private laparoscopic dataset. We report the number of videos per procedure type.