MedCAL-Bench: A Comprehensive Benchmark on Cold-Start Active Learning with Foundation Models for Medical Image Analysis

TL;DR

MedCAL-Bench introduces the first FM-based CSAL benchmark for medical image analysis, evaluating 14 foundation-model feature extractors and 7 sample-selection strategies across 7 datasets covering segmentation and classification. The study finds that most FMs improve over random sampling, with the DINO family excelling in segmentation while general FMs tend to match or surpass medical-specific FMs in several cases; no single FM dominates every dataset. For segmentation, ALPS offers the strongest average performance, whereas RepDiv leads for classification, underscoring the need to tailor sample-selection strategies to the task and dataset. The work emphasizes the pivotal role of FM diversity and feature representation in CSAL efficiency and highlights ongoing gaps, such as the lack of uncertainty-based methods and 3D foundation models, guiding future research in FM-driven CSAL for medical imaging.

Abstract

Cold-Start Active Learning (CSAL) aims to select informative samples for annotation without prior knowledge, which is important for improving annotation efficiency and model performance under a limited annotation budget in medical image analysis. Most existing CSAL methods rely on Self-Supervised Learning (SSL) on the target dataset for feature extraction, which is inefficient and limited by insufficient feature representation. Recently, pre-trained Foundation Models (FMs) have shown powerful feature extraction ability with a potential for better CSAL. However, this paradigm has been rarely investigated, with a lack of benchmarks for comparison of FMs in CSAL tasks. To this end, we propose MedCAL-Bench, the first systematic FM-based CSAL benchmark for medical image analysis. We evaluate 14 FMs and 7 CSAL strategies across 7 datasets under different annotation budgets, covering classification and segmentation tasks from diverse medical modalities. It is also the first CSAL benchmark that evaluates both the feature extraction and sample selection stages. Our experimental results reveal that: 1) Most FMs are effective feature extractors for CSAL, with DINO family performing the best in segmentation; 2) The performance differences of these FMs are large in segmentation tasks, while small for classification; 3) Different sample selection strategies should be considered in CSAL on different datasets, with Active Learning by Processing Surprisal (ALPS) performing the best in segmentation while RepDiv leading for classification. The code is available at https://github.com/HiLab-git/MedCAL-Bench.

Figures (6)

• Figure 1: The workflow of MedCAL-Bench that provides a comprehensive evaluation of both feature extractors and sample selection methods of CSAL on 7 medical datasets for both segmentation and classification tasks. Specifically, we evaluate the effectiveness of foundation models for feature extraction in CSAL, which has not been investigated in existing benchmarks.
• Figure 2: Performance overview of CSAL methods compared with random selection. For each sample selection method on each dataset, the boxplot shows distribution of Dice/accuracy when using different feature extractors (i.e., the 14 FMs).
• Figure 3: The selected samples using MedCLIP-ViT as FM feature extractor and ALPS as the sample selection method for Heart dataset (left:samples with foreground information, right: samples without foreground information).
• Figure 4: The selected samples using MedCLIP-ViT as FM feature extractor and BAL as the sample selection method for Heart dataset (left:samples with foreground information, right: samples without foreground information).
• Figure 5: The selected samples using MedCLIP-ViT as FM feature extractor and ALPS as the sample selection method for Spleen dataset (left:samples with foreground information, right: samples without foreground information).