MILD: Modeling the Instance Learning Dynamics for Learning with Noisy Labels

TL;DR

This work tackles learning with noisy labels by modeling per-instance learning dynamics to distinguish clean from corrupted data. It introduces memorization and forgetting as key signals, combines them into a selection metric, and uses a two-component Weibull mixture to identify a clean subset in iterative rounds. The method can be augmented with semi-supervised techniques to leverage mislabeled data as unlabeled information, yielding superior performance on synthetic and web-noise benchmarks. The results demonstrate that leveraging instance-wise learning dynamics with Weibull-based screening yields robust improvements, with practical impact for scalable learning under label noise.

Abstract

Despite deep learning has achieved great success, it often relies on a large amount of training data with accurate labels, which are expensive and time-consuming to collect. A prominent direction to reduce the cost is to learn with noisy labels, which are ubiquitous in the real-world applications. A critical challenge for such a learning task is to reduce the effect of network memorization on the falsely-labeled data. In this work, we propose an iterative selection approach based on the Weibull mixture model, which identifies clean data by considering the overall learning dynamics of each data instance. In contrast to the previous small-loss heuristics, we leverage the observation that deep network is easy to memorize and hard to forget clean data. In particular, we measure the difficulty of memorization and forgetting for each instance via the transition times between being misclassified and being memorized in training, and integrate them into a novel metric for selection. Based on the proposed metric, we retain a subset of identified clean data and repeat the selection procedure to iteratively refine the clean subset, which is finally used for model training. To validate our method, we perform extensive experiments on synthetic noisy datasets and real-world web data, and our strategy outperforms existing noisy-label learning methods.

Figures (2)

• Figure 1: The overview of each sample selection round. The input dataset at left is noisy like the truck image with cat label. To model learning dynamic for each instance, we record the learning status at each epoch as a sequence, where gray and black squares indicate misclassified and memorized. The $p$ is the number of epochs for current round. Note that the total length of green and red arrows indicate the difficulties of memorization and forgetting. Then we compute the selection metric based on the difficulties. Finally, we identify and select the clean data, and then use them for subsequent round.
• Figure 2: The visualization of metric distribution and mixture model on CIFAR-100 symmetric 20% noise.