Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy

TL;DR

This work tackles the data scarcity and heterogeneity challenges in GI endoscopy by introducing EndoExtend24, a large, multi-dataset GIE image collection with dynamic pathology mappings and strict patient-split integrity. It then demonstrates a domain-adaptive pre-training strategy, fine-tuning EVA-02—an ImageNet-22k pre-trained ViT–on EndoExtend24, followed by CE24-specific training, achieving a macro AUC of $0.993$ and a balanced accuracy of $0.916$ on CE24 validation, and securing third place in the Capsule Endoscopy 2024 Challenge. The approach shows that domain-adaptive self-supervised pre-training on a rich, standardized GI endoscopy dataset can significantly improve downstream diagnostic performance, even in highly imbalanced, multi-class settings. The work also emphasizes dataset interoperability and leakage prevention, underscoring the practical impact of robust data preparation in clinical AI pipelines.

Abstract

Video capsule endoscopy has transformed gastrointestinal endoscopy (GIE) diagnostics by offering a non-invasive method for capturing detailed images of the gastrointestinal tract, enabling early disease detection. However, its potential is limited by the sheer volume of images generated during the imaging procedure, which can take anywhere from 6-8 hours and often produce up to 1 million images, necessitating automated analysis. Additionally, the variability of these images, combined with the need for expert annotations and the scarcity of large, high-quality labeled datasets, constrains the effectiveness of current medical image analysis models. To address this, we introduce a novel large GIE dataset, called EndoExtend24, created by merging ten existing public and private datasets, ensuring patient integrity across splits. EndoExtend24 includes over 226,000 labeled images, as well as dynamic class mappings, which allow unified training across datasets with differing labeling granularity, supporting up to 123 distinct pathological findings. Further, we propose to leverage domain adaptive pre-training of foundation models trained with self-supervision on generic image data, to adapt them to the task of GIE medical image diagnosis. Specifically, the EVA-02 model, which is based on the ViT architecture and trained on ImageNet-22k with masked image modeling (using EVA-CLIP as a MIM teacher), is pre-trained on the EndoExtend24 dataset to achieve domain adaptation, and finally trained on the Capsule Endoscopy 2024 Challenge dataset. Our model demonstrates robust performance, securing third place in the Capsule Endoscopy 2024 Challenge. We achieved a macro AUC of 0.762 and a balanced accuracy of 37.1% on the test set. These results emphasize the effectiveness of our domain-adaptive pre-training approach and the enriched EndoExtend24 dataset in advancing gastrointestinal endoscopy diagnostics.

Figures (4)

• Figure 1: Multi-class GIE datasets that are part of EndoExtend. Datasets are grouped by their split type and contain images obtained via Video Capsule Endoscopy (VCE), Gastroscopy (GST), and Colonoscopy (COL). *The SEE-AI Project dataset employed a 3-fold CV split in the paper, but did not provide this split publically.
• Figure 2: Distribution of pathological findings across the different datasets included in EndoExtend24 which was utilized during pre-training.
• Figure 3: ROC curves of eva02-base on the downstream task of GIE multi-class classification on the CE24 validation dataset. Left: Individual ROC curves for 10 distinct pathological findings, with corresponding AUC values. Right: Micro- and macro-average ROC curves.
• Figure 4: Confusion matrix of our fine-tuned eva02-base on the CE validation set.