Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis

Abstract

Endoscopic video analysis is essential for early gastrointestinal screening but remains hindered by limited high-quality annotations. While self-supervised video pre-training shows promise, existing methods developed for natural videos prioritize dense spatio-temporal modeling and exhibit motion bias, overlooking the static, structured semantics critical to clinical decision-making. To address this challenge, we propose Focus-to-Perceive Representation Learning (FPRL), a cognition-inspired hierarchical framework that emulates clinical examination. FPRL first focuses on intra-frame lesion-centric regions to learn static semantics, and then perceives their evolution across frames to model contextual semantics. To achieve this, FPRL employs a hierarchical semantic modeling mechanism that explicitly distinguishes and collaboratively learns both types of semantics. Specifically, it begins by capturing static semantics via teacher-prior adaptive masking (TPAM) combined with multi-view sparse sampling. This approach mitigates redundant temporal dependencies and enables the model to concentrate on lesion-related local semantics. Following this, contextual semantics are derived through cross-view masked feature completion (CVMFC) and attention-guided temporal prediction (AGTP). These processes establish cross-view correspondences and effectively model structured inter-frame evolution, thereby reinforcing temporal semantic continuity while preserving global contextual integrity. Extensive experiments on 11 endoscopic video datasets show that FPRL achieves superior performance across diverse downstream tasks, demonstrating its effectiveness in endoscopic video representation learning. The code is available at https://github.com/MLMIP/FPRL.

Figures (9)

• Figure 1: (a) Dense spatio-temporal modeling often captures irrelevant semantics due to motion reliance. (b) Our hierarchical approach isolates static from dynamic context, extracting appearance features required for diagnosis.
• Figure 2: The pipeline of the proposed FPRL. For static semantic focus, we employ a teacher-prior adaptive masking strategy coupled with pixel reconstruction. For contextual semantic perception, we use cross-view masked feature completion with latent spatial alignment and an attention-guided temporal prediction module with contrastive learning.
• Figure 3: Qualitative results of segmentation and detection tasks. The segmentation results (green, red, and yellow regions represent the true positive, false positive, and false negative, respectively) on the left are from the CVC-12k dataset, while the detection results on the right are from the KUMC dataset.
• Figure 4: Example frames of the 7 pre-training datasets used in this work.
• Figure 5: Example frames of the 4 downstream datasets used in this work.