Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

Denis Huseljic, Paul Hahn, Marek Herde, Christoph Sandrock, Bernhard Sick

Abstract

Active learning (AL) aims to reduce annotation costs while maximizing model performance by iteratively selecting valuable instances. While foundation models have made it easier to identify these instances, existing selection strategies still lack robustness across different models, annotation budgets, and datasets. To highlight the potential weaknesses of existing AL strategies and provide a reference point for research, we explore oracle strategies, i.e., strategies that approximate the optimal selection by accessing ground-truth information unavailable in practical AL scenarios. Current oracle strategies, however, fail to scale effectively to large datasets and complex deep neural networks. To tackle these limitations, we introduce the Best-of-Strategy Selector (BoSS), a scalable oracle strategy designed for large-scale AL scenarios. BoSS constructs a set of candidate batches through an ensemble of selection strategies and then selects the batch yielding the highest performance gain. As an ensemble of selection strategies, BoSS can be easily extended with new state-of-the-art strategies as they emerge, ensuring it remains a reliable oracle strategy in the future. Our evaluation demonstrates that i) BoSS outperforms existing oracle strategies, ii) state-of-the-art AL strategies still fall noticeably short of oracle performance, especially in large-scale datasets with many classes, and iii) one possible solution to counteract the inconsistent performance of AL strategies might be to employ an ensemble-based approach for the selection.

BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

Abstract

Active learning (AL) aims to reduce annotation costs while maximizing model performance by iteratively selecting valuable instances. While foundation models have made it easier to identify these instances, existing selection strategies still lack robustness across different models, annotation budgets, and datasets. To highlight the potential weaknesses of existing AL strategies and provide a reference point for research, we explore oracle strategies, i.e., strategies that approximate the optimal selection by accessing ground-truth information unavailable in practical AL scenarios. Current oracle strategies, however, fail to scale effectively to large datasets and complex deep neural networks. To tackle these limitations, we introduce the Best-of-Strategy Selector (BoSS), a scalable oracle strategy designed for large-scale AL scenarios. BoSS constructs a set of candidate batches through an ensemble of selection strategies and then selects the batch yielding the highest performance gain. As an ensemble of selection strategies, BoSS can be easily extended with new state-of-the-art strategies as they emerge, ensuring it remains a reliable oracle strategy in the future. Our evaluation demonstrates that i) BoSS outperforms existing oracle strategies, ii) state-of-the-art AL strategies still fall noticeably short of oracle performance, especially in large-scale datasets with many classes, and iii) one possible solution to counteract the inconsistent performance of AL strategies might be to employ an ensemble-based approach for the selection.
Paper Structure (27 sections, 6 equations, 16 figures, 13 tables, 1 algorithm)

This paper contains 27 sections, 6 equations, 16 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notation
  4. A Formalization of Performance-based Active Learning
  5. An Efficient Oracle Strategy for Deep Neural Networks
  6. Batch Selection
  7. Performance Estimation
  8. Retraining
  9. Comparison of Time Complexity
  10. Empirical Evaluation of BoSS: Oracle-Level and State-of-the-art AL Comparisons
  11. Experimental Setup
  12. Benchmark Results
  13. Analytical Evaluation: Ablations and Sensitivity Analyses
  14. Selection of Candidate Batches
  15. Estimation of Performance
  16. ...and 12 more sections

Figures (16)

  • Figure 1: Accuracy improvement over random sampling for BoSS and state-of-the-art selection strategies using DINOv2-ViT-S/14.
  • Figure 2: Relative learning curves of oracle strategies with aligned runtimes using DINOv2-ViT-S/14.
  • Figure 3: Relative learning curves achieved by BoSS and state-of-the-art selection strategies at each annotation cycle for different pretrained models.
  • Figure 4: Average relative pick frequency of AL strategies by BoSS across cycles averaged over all datasets.
  • Figure 5: Relative learning curves of BoSS with naively selected candidate batches vs. our algorithm.
  • ...and 11 more figures