PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

TL;DR

The paper tackles the data scarcity in active learning by proposing ActiveKD, which distills knowledge from vision language models to train compact task specific models with limited annotations. A key insight is that VLM teachers exhibit structured prediction bias that clusters predictions in the probability simplex; the authors model this as a finite union of $K$ balls with centers $\mu_k$ and radii $r_k$ and show that this bias propagates to the student via KD. To leverage this, they introduce Probabilistic CoreSet PCoreSet that selects samples by maximizing coverage in the probability space using distance $d(x, x') = ||f_r(x) - f_r(x')||_2$, enabling more effective transfer under tight budgets. Empirical results on 11 datasets show consistent gains for ActiveKD across selection strategies (e.g., +29.07% on ImageNet) and that PCoreSet ranks first in about 88% of zero shot / few shot settings across five students and three teachers.

Abstract

Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs-i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves performance across selection methods (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, PCoreSet ranks first in 64/73 settings (approximately 87.7%) across 5 student and 3 teacher networks, always achieving the best performance except for first 2 AL rounds. Our code is available at https://github.com/erjui/PCoreSet.

Figures (25)

• Figure 1: (Left): The proposed ActiveKD framework consistently improves the final-round accuracy of all selection methods, while PCoreSetfurther outperforms baselines when combined with ActiveKD. (Right): ActiveKD consistently improves the performance of PCoreSet across active learning rounds (No Distill vs. Zero-shot Distill), with further gains when using few-shot teachers (Few-shot Distill).
• Figure 2: An overview of ActiveKD.
• Figure 3: (Left): Teacher model prediction biases ($(\mu_1, r_1), \ldots, (\mu_k, r_k)$) are transferred to student models via distillation, where $\hat{\mu}_k=\mu_k+y_c$ and $\hat{r}_k=(1-\lambda)r_k$ ($\mu$ denotes centroids, $r$ denotes radii). (Right):PCoreSet selects samples maximizing distance to labeled points in probability simplex $\Delta^{C-1}$, uncovering underrepresented regions.
• Figure 4: Visualization of prediction bias and its propagation. (a): hard prediction bias of the teacher; (b): soft prediction bias of the teacher (left) and the student (right) after distillation; and (c): the bias propagated from the teacher to the student, quantified by KNN loss ($\ell_2$) across different numbers of clusters ($K$).
• Figure 5: Results on ImageNet and the average over 8 or 10 datasets across 5 student and 3 teacher architectures under ActiveKD (Zero-Shot) with either 8 or 16 rounds. We report the mean and 95% CI over five runs. PCoreSet achieves the best performance in 64/73 settings ($\approx$87.7%)

Theorems & Definitions (9)

• Definition 1: Structured prediction bias
• Proposition 1: Bias propagation through KD
• Theorem 1: Optimal Distribution for Knowledge Distillation
• Theorem 2: Optimal Prediction as Linear Combination
• Definition 2: Structured prediction bias
• Proposition 2: Bias propagation through KD
• proof
• Corollary 1: Student Bias Inheritance
• proof