Efficient Adaptive Label Refinement for Label Noise Learning

TL;DR

This work tackles the problem of deep models memorizing noisy labels by decoupling robust learning from label denoising. It introduces Adaptive Label Refinement (ALR), a two-phase method that first trains with hard labels and then progressively refines targets to soft distributions via temporal ensembling, supplemented by a small entropy regularization to emphasize learning from clean samples. ALR does not require prior knowledge of noise distributions or external clean data and demonstrates state-of-the-art performance on CIFAR-10/100 with artificial noise and real-world noisy datasets (ANIMAL-10N, Clothing1M, WebVision) along with comprehensive ablations. The approach is simple, effective, and broadly applicable, offering a practical solution for robust learning under label noise.

Abstract

Deep neural networks are highly susceptible to overfitting noisy labels, which leads to degraded performance. Existing methods address this issue by employing manually defined criteria, aiming to achieve optimal partitioning in each iteration to avoid fitting noisy labels while thoroughly learning clean samples. However, this often results in overly complex and difficult-to-train models. To address this issue, we decouple the tasks of avoiding fitting incorrect labels and thoroughly learning clean samples and propose a simple yet highly applicable method called Adaptive Label Refinement (ALR). First, inspired by label refurbishment techniques, we update the original hard labels to soft labels using the model's predictions to reduce the risk of fitting incorrect labels. Then, by introducing the entropy loss, we gradually `harden' the high-confidence soft labels, guiding the model to better learn from clean samples. This approach is simple and efficient, requiring no prior knowledge of noise or auxiliary datasets, making it more accessible compared to existing methods. We validate ALR's effectiveness through experiments on benchmark datasets with artificial label noise (CIFAR-10/100) and real-world datasets with inherent noise (ANIMAL-10N, Clothing1M, WebVision). The results show that ALR outperforms state-of-the-art methods.

Figures (9)

• Figure 1: t-SNE visualizations in plots (a) and (b) illustrate the relationships among clean samples in the CIFAR-10 training set after training the model with the pure label refurbishment (LR) method under 20% and 80% symmetric label noise.
• Figure 2: Plots (a) and (b) illustrate the training and testing accuracies of DNNs employing cross-entropy loss on the CIFAR-10/100 datasets, each with 60% symmetric label noise.
• Figure 3: The architecture of the ALR training framework. In the warm-up phase, the model is trained using cross-entropy loss $\mathcal{L}_{ce}$ with the original noisy labels ($\mathbf{y}$). In the label refinement phase, the noisy labels are replaced with soft labels, and an entropy loss term $\mathcal{L}_{\text{e}}$ is incorporated to assist in training the model. At each epoch, the soft label $\mathbf{t}(e)$ is updated by blending the soft label from the previous epoch $\mathbf{t}(e-1)$ and the current prediction $\mathbf{p}(e)$, using a weighted sum.
• Figure 4: We studied the memorization phenomenon of DNNs on CIFAR-10 using CE and ALR under 40% symmetric noise. Plots (a) and (c) illustrate the proportion of correct predictions (in blue) and incorrect predictions (in red) on clean samples across different training phases. Plots (b) and (d) show the proportion of correct predictions (in blue), incorrect predictions (i.e., predictions that are neither equal to the true label nor the erroneous label, in red), and memorization (i.e., predictions that match the erroneous label, in green) on mislabeled samples across different training phases.
• Figure 5: 2D t-SNE visualization of clean samples in the CIFAR-10 training set for the simplest label refurbishment method (LR) and ALR under different levels of label noise.

Theorems & Definitions (2)

• Theorem 1
• proof