Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Records

Yili He, Yan Zhu, Peiyao Fu, Ruijie Yang, Tianyi Chen, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang

TL;DR

Endo-CLIP tackles image-text alignment in raw colonoscopy records by addressing background noise, complex medical terminology, and multi-polyp ambiguity through a three-stage framework: cleansing to purify frames, attunement to align single-polyp morphology with textual descriptions, and unification to resolve multi-polyp contexts with cross-attention. It leverages LLM-derived morphological attributes and patient-level aggregation to produce robust polyp representations that perform well in zero-shot and few-shot settings. Evaluations on a large dataset and the EndoReport50 benchmark show state-of-the-art performance in polyp detection and malignancy classification, indicating strong potential for clinical deployment and improved endoscopic analysis. Overall, Endo-CLIP advances clinically relevant image-text pre-training for endoscopy by integrating multimodal cues and lesion-level context to enhance generalization and interpretability.

Abstract

Pre-training on image-text colonoscopy records offers substantial potential for improving endoscopic image analysis, but faces challenges including non-informative background images, complex medical terminology, and ambiguous multi-lesion descriptions. We introduce Endo-CLIP, a novel self-supervised framework that enhances Contrastive Language-Image Pre-training (CLIP) for this domain. Endo-CLIP's three-stage framework--cleansing, attunement, and unification--addresses these challenges by (1) removing background frames, (2) leveraging large language models to extract clinical attributes for fine-grained contrastive learning, and (3) employing patient-level cross-attention to resolve multi-polyp ambiguities. Extensive experiments demonstrate that Endo-CLIP significantly outperforms state-of-the-art pre-training methods in zero-shot and few-shot polyp detection and classification, paving the way for more accurate and clinically relevant endoscopic analysis.

Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Records

TL;DR

Endo-CLIP tackles image-text alignment in raw colonoscopy records by addressing background noise, complex medical terminology, and multi-polyp ambiguity through a three-stage framework: cleansing to purify frames, attunement to align single-polyp morphology with textual descriptions, and unification to resolve multi-polyp contexts with cross-attention. It leverages LLM-derived morphological attributes and patient-level aggregation to produce robust polyp representations that perform well in zero-shot and few-shot settings. Evaluations on a large dataset and the EndoReport50 benchmark show state-of-the-art performance in polyp detection and malignancy classification, indicating strong potential for clinical deployment and improved endoscopic analysis. Overall, Endo-CLIP advances clinically relevant image-text pre-training for endoscopy by integrating multimodal cues and lesion-level context to enhance generalization and interpretability.

Abstract

Pre-training on image-text colonoscopy records offers substantial potential for improving endoscopic image analysis, but faces challenges including non-informative background images, complex medical terminology, and ambiguous multi-lesion descriptions. We introduce Endo-CLIP, a novel self-supervised framework that enhances Contrastive Language-Image Pre-training (CLIP) for this domain. Endo-CLIP's three-stage framework--cleansing, attunement, and unification--addresses these challenges by (1) removing background frames, (2) leveraging large language models to extract clinical attributes for fine-grained contrastive learning, and (3) employing patient-level cross-attention to resolve multi-polyp ambiguities. Extensive experiments demonstrate that Endo-CLIP significantly outperforms state-of-the-art pre-training methods in zero-shot and few-shot polyp detection and classification, paving the way for more accurate and clinically relevant endoscopic analysis.
Paper Structure (11 sections, 5 equations, 3 figures, 2 tables)

This paper contains 11 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Model Overview
  4. Cleansing Stage: Image Purification
  5. Attunement Stage: Single-Polyp Semantic Alignment
  6. Unification Stage: Multi-Polyp Contextual Integration
  7. Experiment
  8. Implementation Details
  9. Downstream Task Setup
  10. Experimental Results
  11. Conclusion

Figures (3)

  • Figure 1: Challenges of pre-training on colonoscopy records and the Endo-CLIP solution.
  • Figure 2: Overview of Endo-CLIP: a progressive self-supervised pre-training framework that refines endoscopic image–text alignment.
  • Figure 3: t-SNE van2008visualizing visualization of feature distributions for malignancy classification. Each point represents a polyp image in the feature space, with green circles for benign cases and red triangles for malignant cases.