Deep Active Learning in the Open World

TL;DR

This work tackles open-world learning under long-tail distributions and limited annotation budgets by introducing ALOE, a two-stage active learning algorithm that first enforces diversity via clustering and then prioritizes potential unknowns using GradNorm-based OOD scoring. By empirically evaluating on CIFAR100-LT, ImageNet-LT, and Places365-LT, ALOE consistently outperforms standard baselines in balanced accuracy and accelerates novel class discovery, notably achieving up to $70\%$ annotation savings on ImageNet-LT. A key insight is the tradeoff between improving known-class accuracy and discovering new classes, which motivates future research into dynamic exploration-exploitation strategies. The approach offers practical benefits for open-world systems where the total class set is unknown and labeling is costly, enabling more robust and scalable model adaptation.

Abstract

Machine learning models deployed in open-world scenarios often encounter unfamiliar conditions and perform poorly in unanticipated situations. As AI systems advance and find application in safety-critical domains, effectively handling out-of-distribution (OOD) data is crucial to building open-world learning systems. In this work, we introduce ALOE, a novel active learning algorithm for open-world environments designed to enhance model adaptation by incorporating new OOD classes via a two-stage approach. First, diversity sampling selects a representative set of examples, followed by energy-based OOD detection to prioritize likely unknown classes for annotation. This strategy accelerates class discovery and learning, even under constrained annotation budgets. Evaluations on three long-tailed image classification benchmarks demonstrate that ALOE outperforms traditional active learning baselines, effectively expanding known categories while balancing annotation cost. Our findings reveal a crucial tradeoff between enhancing known-class performance and discovering new classes, setting the stage for future advancements in open-world machine learning.

Figures (8)

• Figure 1: Visualization of highest OOD score unlabeled examples after training on three known classes in CIFAR-100. As we can see, relying only on OOD scores for selection will encourage annotation of unlabeled examples in a few (unknown) classes.
• Figure 2: Illustration of our algorithm ALOE in Algorithm \ref{['alg:aloe']}. Starting with a few labeled examples, the algorithm clusters all examples, ideally by their underlying classes. Each example's OOD score is calculated. In the bottom right plot, larger dots mean larger OOD score. Clusters with higher OOD ratios are prioritized for sampling to identify new classes. Labeled examples are then added to the training set, and the process iterates, expanding the labeled pool with each batch.
• Figure 3: With different initial number of annotated classes, balanced test accuracy and number of annotated classes of CIFAR100-LT. (All subfigures share the same legend of the left bottom one. Shaded region represent the standard error conducted across four trials.)
• Figure 4: Ablation experiment for using different OOD scores $\Omega(x, f)$ in ALOE on CIFAR100-LT. (Shaded region represents the standard error conducted across four trials.)
• Figure 5: Ablation experiment for clustering methods on CIFAR100-LT with GradNorm OOD score. (Shaded region represent the standard error conducted across four trials.)