Combating Label Noise With A General Surrogate Model For Sample Selection

TL;DR

This work tackles label noise in web-sourced image data by introducing a training-free sample-selection mechanism that leverages the vision-language model CLIP to identify clean training samples via prediction confidence and prompt-consistency criteria. To counteract biases introduced by CLIP and maintain robustness, it couples this sampling with a Noise-Aware Balanced Margin Adaptive (NABM) loss that integrates a transition matrix and class-frequency priors to adjust logits and emphasize hard, underrepresented cases using a focal formulation. The approach demonstrates significant improvements over strong baselines on multiple real-world and synthetic noisy datasets, while ensuring CLIP is not needed during inference. Overall, the paper presents a practical, general framework for mitigating label noise by combining external knowledge with a tailored margins-based loss.

Abstract

Modern deep learning systems are data-hungry. Learning with web data is one of the feasible solutions, but will introduce label noise inevitably, which can hinder the performance of deep neural networks. Sample selection is an effective way to deal with label noise. The key is to separate clean samples based on some criterion. Previous methods pay more attention to the small loss criterion where small-loss samples are regarded as clean ones. Nevertheless, such a strategy relies on the learning dynamics of each data instance. Some noisy samples are still memorized due to frequently occurring corrupted learning patterns. To tackle this problem, a training-free surrogate model is preferred, freeing from the effect of memorization. In this work, we propose to leverage the vision-language surrogate model CLIP to filter noisy samples automatically. CLIP brings external knowledge to facilitate the selection of clean samples with its ability of text-image alignment. Furthermore, a margin adaptive loss is designed to regularize the selection bias introduced by CLIP, providing robustness to label noise. We validate the effectiveness of our proposed method on both real-world and synthetic noisy datasets. Our method achieves significant improvement without CLIP involved during the inference stage.

Figures (8)

• Figure 1: Some small-loss noisy samples that share similar visual patterns are memorized by DNNs. They are misidentified as clean samples by the small loss criterion. With the help of the powerful open-vocabulary vision-language model CLIP, these samples can be further filtered out potentially. With cleaner training samples, the classification performance is further boosted.
• Figure 2: The overall framework is presented. We leverage the open-vocabulary vision-language surrogate model CLIP to select clean samples. The annotated confidence is predicted by CLIP corresponding to its noisy label. Here, red denotes noisy samples treated by CLIP and green denotes clean ones. Then, combined with the transition matrix and the class frequency prior, we propose a noise-aware balanced margin adaptive loss to mitigate the overconfidence effect and the class imbalanced issue.
• Figure 3: Examples of the selected or filtered images by the prompt consistency strategy. red denotes the annotated noisy label is wrong and green represents the annotated noisy label is consistent with its true label.
• Figure 4: \ref{['fig:ab_threshold']}: Ablation study on the effect of selection threshold. We plot the number of training samples and top-1 accuracy (%) on WebVision and ImageNet with different thresholds. \ref{['fig:ab_margin']}: Ablation study on the effect of the noise-aware margin. We report top-1 and top-5 test accuracy (%) on both WebVision and ImageNet.
• Figure 5: Confidence distribution comparison between GMM and CLIP on WebVision. We divide confidence into 10 intervals, each with a range of 0.1, and count the image amount for each interval.