Active Learning with a Noisy Annotator

TL;DR

This work tackles active learning under noisy annotators by introducing Noise-Aware Active Sampling (NAS), a framework that augments greedy, coverage-based query strategies with a low-budget noise-filtering module. NAS partitions labeled data into clean and noisy sets, preserves clean examples for query selection, and resamples from regions previously misrepresented by noisy labels, typically using a batch size $b = C$ (the number of classes). Through ProbCover-based NPC and extensions like Weighted NPC and noise dropout, NAS demonstrates robust performance gains across symmetric, asymmetric, and real-world noise on CIFAR100, ImageNet-50, CIFAR100N, and Clothing1M, using self-supervised representations (e.g., SimCLR, DINOv2). The findings suggest practical improvements for annotation efficiency in noisy environments and point to future work on multi-annotator scenarios and adaptive strategies for even more robust active learning in the low-budget regime.

Abstract

Active Learning (AL) aims to reduce annotation costs by strategically selecting the most informative samples for labeling. However, most active learning methods struggle in the low-budget regime where only a few labeled examples are available. This issue becomes even more pronounced when annotators provide noisy labels. A common AL approach for the low- and mid-budget regimes focuses on maximizing the coverage of the labeled set across the entire dataset. We propose a novel framework called Noise-Aware Active Sampling (NAS) that extends existing greedy, coverage-based active learning strategies to handle noisy annotations. NAS identifies regions that remain uncovered due to the selection of noisy representatives and enables resampling from these areas. We introduce a simple yet effective noise filtering approach suitable for the low-budget regime, which leverages the inner mechanism of NAS and can be applied for noise filtering before model training. On multiple computer vision benchmarks, including CIFAR100 and ImageNet subsets, NAS significantly improves performance for standard active learning methods across different noise types and rates.

Figures (19)

• Figure 1: Overall visualization of our framework for Noise Aware Query Selection (NAS). NAS (illustrated with a dashed orange line) takes as input a query selection strategy $\mathcal{S}$ and a noise-filtering algorithm $\mathcal{A}$. The framework alternates between selecting $b$ samples using $\mathcal{S}$, sending these samples to the annotator, and filtering the noisy samples with $\mathcal{A}$ before selecting the next set of samples.
• Figure 2: Performance of the AUM method in identifying mislabeled data selected by ProbCover in the low-budget regime on CIFAR100 with symmetric noise. Each column represents a different expected number of clean samples per class ($\mathbb{E}[\text{SPC}]$), with the budget given by $\frac{\mathbb{E}[\text{SPC}] \times C}{1-\%\text{noise}}$. Rows show noise precision, recall, and predicted noise ratio. The orange line represents the original AUM, while the blue line represents LowBudgetAUM. Unlike AUM, which predicts most samples as noisy, LowBudgetAUM estimates noise rates more accurately—even with as few as two clean samples per class—while maintaining high precision and recall. Each point shows the mean and standard error across 10 repetitions.
• Figure 3: Framework \ref{['f1']}, results on CIFAR100 and ImageNet-50 with varying symmetric noise levels. The y-axis shows the mean accuracy difference from random query selection. A ResNet-18 model is trained in a fully supervised manner.
• Figure 4: Framework \ref{['f2']}, see caption of Fig. \ref{['fig:supervised_learning_sym_noise']}. Here we evaluate a linear model trained on self-supervised pretrained features.
• Figure 5: Results given different levels of asymmetric noise.