Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learning

TL;DR

The paper tackles the instability of selecting a single active-learning strategy by introducing REFINE, an ensemble method that first progressively filters the unlabeled pool by unioning selections from multiple strategies and then applies a coverage-based final batch selection. The authors provide theoretical guarantees for preserving high-value instances and exponentially shrinking uninformative ones, and demonstrate superior performance across six image datasets and three foundation-model backbones, including a practical audio use case. REFINE's key contribution is a training-free, modular preprocessing step that robustly combines diverse notions of data value and can readily incorporate new strategies. The work suggests significant practical benefits for scalable, strategy-agnostic active learning workflows.

Abstract

Existing active learning (AL) strategies capture fundamentally different notions of data value, e.g., uncertainty or representativeness. Consequently, the effectiveness of strategies can vary substantially across datasets, models, and even AL cycles. Committing to a single strategy risks suboptimal performance, as no single strategy dominates throughout the entire AL process. We introduce REFINE, an ensemble AL method that combines multiple strategies without knowing in advance which will perform best. In each AL cycle, REFINE operates in two stages: (1) Progressive filtering iteratively refines the unlabeled pool by considering an ensemble of AL strategies, retaining promising candidates capturing different notions of value. (2) Coverage-based selection then chooses a final batch from this refined pool, ensuring all previously identified notions of value are accounted for. Extensive experiments across 6 classification datasets and 3 foundation models show that REFINE consistently outperforms individual strategies and existing ensemble methods. Notably, progressive filtering serves as a powerful preprocessing step that improves the performance of any individual AL strategy applied to the refined pool, which we demonstrate on an audio spectrogram classification use case. Finally, the ensemble of REFINE can be easily extended with upcoming state-of-the-art AL strategies.

Figures (13)

• Figure 1: Refine's two-stage selection process. Stage 1: Progressive filtering refines the unlabeled pool through multiple rounds, where each round outputs the union of instances selected by each strategy as the next round's input pool. Stage 2: Coverage-based selection then chooses the final batch from this refined pool, ensuring a selection that accounts for diverse notions of value.
• Figure 2: Pairwise comparison matrix averaged across 3 backbones, 5 datasets, and 10 trials. Element $(i,j)$ corresponds to the proportion of total runs, where strategy $i$ outperforms strategy $j$.
• Figure 3: Relative accuracy learning curves for Refine and baseline strategies across multiple backbones and datasets.
• Figure 4: Pairwise comparison matrix of ensemble AL methods averaged across 3 backbones, 5 datasets, and 10 trials.
• Figure 5: Benefits of progressive filtering (selecting from ${\mathcal{C}}_R$ vs. ${\mathcal{U}}_t$) on CIFAR-100 across different strategies.

Theorems & Definitions (5)

• Theorem 1: Value Preservation Bound
• Definition 1: Uninformative Instance
• Theorem 2: Exponential Reduction of Uninformative Instances
• Theorem 2: Exponential Reduction of Uninformative Instances
• Theorem 3: Concentration of Expected Value