Conformal Cross-Modal Active Learning

Abstract

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.

Figures (14)

• Figure 1: Overview of our proposed AL framework CCMA for image classification by exploring conformal prediction with multimodal uncertainty and diversity for efficient data acquisition. Given labeled data $\mathcal{L}$ and an unlabeled pool $\mathcal{U}$, a frozen vision encoder (student) and a frozen VLM (teacher) serve as feature extractors, while a simple linear classifier is trained on the student features. (1) A selective subpool is formed via CLIP-space clustering. (2–3) Student and teacher posteriors $p_S$, $p_T$ are calibrated into conformal sets $\Gamma_S,\Gamma_T$. (4) Multimodal disagreement $\Omega(x)$ combines entropy and Jensen-Shannon (JS) divergence for uncertainty scoring $\delta(x)$. (5) Top-ranked samples are oversampled and clustered with uncertainty-weighted coverage for the final diverse selection.
• Figure 2: Labels required to reach target accuracies of 80% (blue), 85% (orange), and 90% (green) on Food101 and DomainNet-Real. Lower values indicate higher label efficiency. CCMA consistently reaches each accuracy threshold with fewer labeled samples than uncertainty- and coverage-based baselines, demonstrating improved sample efficiency across both datasets, especially in low-budget regimes.
• Figure 3: CCMA diagnostics in the overlap and the fraction Top-1 disagreement between teacher and student.
• Figure 4: Test mean accuracy over 5 seeds for CCMA with other AL methods on Caltech256 and Caltech101 datasets.
• Figure 5: Effect of oversampling factor $\kappa$.