Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization

TL;DR

This work tackles learning with noisy labels by introducing wrong_event, a simple, stable metric that tracks cumulative misclassifications and remains informative throughout training. By modeling wrong_event with per-class Beta mixtures, the authors derive per-sample cleanliness and difficulty scores, enabling a tuning-free, instance-level dynamic loss weighting. The proposed two-stage framework (Stage 1: obtain a competitive base model; Stage 2: robust training with a weighted loss combining clean, noisy, and difficult components) yields a loss function that adapts to each sample without hyperparameter tuning. Across synthetic and real-world benchmarks, IDO achieves state-of-the-art accuracy and substantially reduces training time, offering scalable, practical improvements for learning under label noise.

Abstract

Recent studies indicate that deep neural networks degrade in generalization performance under noisy supervision. Existing methods focus on isolating clean subsets or correcting noisy labels, facing limitations such as high computational costs, heavy hyperparameter tuning process, and coarse-grained optimization. To address these challenges, we propose a novel two-stage noisy learning framework that enables instance-level optimization through a dynamically weighted loss function, avoiding hyperparameter tuning. To obtain stable and accurate information about noise modeling, we introduce a simple yet effective metric, termed wrong event, which dynamically models the cleanliness and difficulty of individual samples while maintaining computational costs. Our framework first collects wrong event information and builds a strong base model. Then we perform noise-robust training on the base model, using a probabilistic model to handle the wrong event information of samples. Experiments on five synthetic and real-world LNL benchmarks demonstrate our method surpasses state-of-the-art methods in performance, achieves a nearly 75% reduction in computational time and improves model scalability.

Figures (8)

• Figure 1: A bar chart illustrates the distribution of wrong events. In terms of sample cleanness, clean samples show lower wrong event, while noisy samples tend to have higher wrong event. In terms of sample difficulty, samples located at the extremes of the distribution represent easy samples, far from the decision boundaries (e.g., cat and bird in the figure). In contrast, samples located in the central depression of the distribution are often near the decision boundaries of similar classes, representing hard samples (e.g., cat and leopard in the figure).
• Figure 2: A comparison between wrong event and loss. The baseline model is ResNet-18 trained on CIFAR-10 with 40% symmetric noise training for 100 epochs. We show the loss distributions (the first row) and wrong event distribution (the second row) during training. The four columns, (a) (b) (c) (d), represent the distributions at epoch 20, 60, 100 and the entire training phase. (a), (b), and (c) represent the early, middle, and late stages, respectively. In (d), the heavy lines represent the mean value and the shaded areas are the interquartile ranges. Since wrong events are monotonically increasing based on historical statistics instead of current model prediction, when model overfits the dataset, wrong event values for all samples do not change, rather than converging to zero as loss values typically do. As a result, wrong event can clearly separate noisy data and clean data in all training stages, even if the model fits noise.
• Figure 3: Illustration of the proposed IDO framework. The training process is divided into two stages. In the first stage, prior knowledge, i.e., coarse distribution of wrong event, is obtained, and a base model that owns basic discrimination capability is captured. In the second stage, robust noise learning is performed. By using BMM, we obtain both cleanliness and difficulty information for individual samples, enabling instance-level dynamic optimization. The sample's wrong event information and the model's classification capability mutually benefit each other, leading to high improvement.
• Figure 4: Comparison with state-of-the-art LNL algorithms in effectiveness and efficiency, using pre-trained ResNet50 on the A100 80GB GPU and CIFAR-100 dataset with 20% symmetric noise.
• Figure 5: The baseline model is ResNet-18 trained on CIFAR-10 in 40% asymmetric noise for 100 epochs. We show the loss distributions (the first row) and wrong event distribution (the second row) during training. The four columns, (a) (b) (c) (d), represent the distributions at epoch 10, 50, 100 and the entire training phase. In (d), the heavy lines represent the median value and the shaded areas are the interquartile ranges, respectively. It is clear that wrong event can clearly separate noisy data and clean data, even if the model fits noise.