Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

Zhao Wang, Chang Liu, Shaoting Zhang, Qi Dou

TL;DR

Endo-FM addresses the lack of foundation models for endoscopy video analysis by introducing a ViT-based video transformer backbone pre-trained with a novel self-supervised, spatial-temporal matching framework. The model learns robust, view-invariant representations through a teacher–student scheme that uses global and local spatial-temporal views and two matching losses, L_cv and L_dm, augmented with dynamic positional encoding. A large-scale dataset combining public and private endoscopy videos (over 33K clips, up to 5 million frames) enables strong downstream performance across disease diagnosis, segmentation, and detection, outperforming SOTA self-supervised pre-training and adapter-based transfer learning. The work demonstrates the practicality of domain-specific foundation models for medical video analysis and provides public code and data to advance research in endoscopic video understanding.

Abstract

Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

TL;DR

Endo-FM addresses the lack of foundation models for endoscopy video analysis by introducing a ViT-based video transformer backbone pre-trained with a novel self-supervised, spatial-temporal matching framework. The model learns robust, view-invariant representations through a teacher–student scheme that uses global and local spatial-temporal views and two matching losses, L_cv and L_dm, augmented with dynamic positional encoding. A large-scale dataset combining public and private endoscopy videos (over 33K clips, up to 5 million frames) enables strong downstream performance across disease diagnosis, segmentation, and detection, outperforming SOTA self-supervised pre-training and adapter-based transfer learning. The work demonstrates the practicality of domain-specific foundation models for medical video analysis and provides public code and data to advance research in endoscopic video understanding.

Abstract

Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.
Paper Structure (11 sections, 3 equations, 3 figures, 2 tables)

This paper contains 11 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Video Transformer for Spatial-Temporal Encoding
  4. Self-supervised Pre-train via Spatial-Temporal Matching
  5. Overall Optimization Objective and Pre-training Specifics
  6. Experiment
  7. Datasets and Downstream Setup
  8. Comparison with State-of-the-art Methods
  9. Analytical Studies
  10. Conclusion and Discussion
  11. Acknowledgements.

Figures (3)

  • Figure 1: Illustration of our proposed Endo-FM. We build a video transformer model and design a self-supervised pre-training approach.
  • Figure 2: Example frames of the 10 pre-train and downstream datasets used in this work.
  • Figure 3: Ablations on PolypDiag: (a) components analysis; (b) different combinations of views; (c) number of global and local views; (d) length of local views.