SAMOSA: Sharpness Aware Minimization for Open Set Active learning

TL;DR

SAMOSA tackles the challenge of open-set active learning by prioritizing atypical samples that lie near decision boundaries through a discrepancy-driven framework between SGD and Sharpness Aware Minimization (SAM). It introduces SAMIS-P, a ground-truth-free proxy $S(x)=||f_{SAM}(x)-f_{SGD}(x)||_1$ to identify informative atypical data, and a two-phase SAMOSA algorithm that first filters out unknowns with a $|K|+1$ distinguisher and then selects high-SAMOS-P samples to label. The approach achieves state-of-the-art accuracy on CIFAR10, CIFAR100, and TinyImageNet across 11 rounds, with robustness to label noise and competitive compute relative to baselines. By focusing on sample atypicality and decision-boundary proximity, SAMOSA demonstrates that sample quality can outperform sheer quantity in OSAL, offering practical gains for reducing labeling costs in real-world, open-world setups.

Abstract

Modern machine learning solutions require extensive data collection where labeling remains costly. To reduce this burden, open set active learning approaches aim to select informative samples from a large pool of unlabeled data that includes irrelevant or unknown classes. In this context, we propose Sharpness Aware Minimization for Open Set Active Learning (SAMOSA) as an effective querying algorithm. Building on theoretical findings concerning the impact of data typicality on the generalization properties of traditional stochastic gradient descent (SGD) and sharpness-aware minimization (SAM), SAMOSA actively queries samples based on their typicality. SAMOSA effectively identifies atypical samples that belong to regions of the embedding manifold close to the model decision boundaries. Therefore, SAMOSA prioritizes the samples that are (i) highly informative for the targeted classes, and (ii) useful for distinguishing between targeted and unwanted classes. Extensive experiments show that SAMOSA achieves up to 3% accuracy improvement over the state of the art across several datasets, while not introducing computational overhead. The source code of our experiments is available at: https://anonymous.4open.science/r/samosa-DAF4

Figures (19)

• Figure 1: Limitations of uncertainty-based sampling. (a-b) Visualization of: (a) lowest entropy samples that were misclassified for the class 'ship' on CIFAR10 with 30% mismatch at round 1. These are atypical, informative examples that will be ignored by the entropy criterion, but will be picked by our method; (b) batches of typical ships selected using entropy. (c-d) Embedding manifold visualization of the data points used to train (in green) against the queried sampled for labeling (in red) at round 5 over different approaches. Rather than relying on uncertainty criterion which collect samples scattered across the classes clusters (c), our approach selects data points which fall close to decision boundaries, showcasing their informative nature (d). (e) Classification performance comparison on CIFAR100 with 40% mismatch ratio. SAMOSA overcomes state-of-the-art by $3\%$.
• Figure 2: Evolution of the impact of SAMIS-P scores of training samples on their embeddings across training epochs. Samples corresponding to high scores (red points) occupy critical positions relative to other data points, especially during the central epochs of the training where they are more concentrated around the boundary areas between the class clusters.
• Figure 3: Entropy distribution of misclassified samples queried using high entropy (red) and high SAMIS (blue). Numbers in the legend indicate the number of misclassified samples. SAMOSA queries more samples the model misclassify with high confidence, especially in earlier rounds.
• Figure 4: Classification performance comparison on CIFAR10 (first row), CIFAR100 (second row) and Tiny-Imagenet (third row) with 20% (first column), 30% (second column) and 40% (third column) mismatch ratio. SAMOSA overcomes state-of-the-art by up to $2\%$ on CIFAR10, $3\%$ on CIFAR100 and $2\%$ on TinyImageNet.
• Figure 5: Accuracy, recall, and precision of bucketed SAMOSA over the CIFAR10 dataset with $40\%$ mismatch ratio. As bucket number increases, the precision drops while the accuracy increases.

Theorems & Definitions (9)

• Theorem 2.1
• Lemma A.2
• Lemma A.3
• Lemma A.4
• Definition A.5: Our Data Setup
• Theorem A.6: formal statement of Theorem \ref{['theoremInformal']}
• proof
• proof
• proof