Tackling Noisy Labels with Network Parameter Additive Decomposition

TL;DR

This work tackles noisy-label learning in over-parameterized networks by introducing additive parameter decomposition, splitting weights into σ (ideal, memorizing clean data) and γ (noise, memorizing mislabeled data). A time-varying objective L_F = L + β_1(t)Δσ + β_2(t)∥γ∥_2, with Δσ = ∥σ − σ_{t-1}∥_2, steers the training trajectory so σ rapidly fits clean data early while γ absorbs label noise later, and inference uses only σ. The authors demonstrate TNLPAD's effectiveness on diverse simulated and real-world noisy datasets, with extensive ablations showing the importance of the decomposition and the dynamic constraints, and show compatibility with semi-supervised methods. The approach provides a practical, architecture-agnostic mechanism to improve generalization under label noise, potentially extendable to other challenging domains like NLP.

Abstract

Given data with noisy labels, over-parameterized deep networks suffer overfitting mislabeled data, resulting in poor generalization. The memorization effect of deep networks shows that although the networks have the ability to memorize all noisy data, they would first memorize clean training data, and then gradually memorize mislabeled training data. A simple and effective method that exploits the memorization effect to combat noisy labels is early stopping. However, early stopping cannot distinguish the memorization of clean data and mislabeled data, resulting in the network still inevitably overfitting mislabeled data in the early training stage.In this paper, to decouple the memorization of clean data and mislabeled data, and further reduce the side effect of mislabeled data, we perform additive decomposition on network parameters. Namely, all parameters are additively decomposed into two groups, i.e., parameters $\mathbf{w}$ are decomposed as $\mathbf{w}=\bmσ+\bmγ$. Afterward, the parameters $\bmσ$ are considered to memorize clean data, while the parameters $\bmγ$ are considered to memorize mislabeled data. Benefiting from the memorization effect, the updates of the parameters $\bmσ$ are encouraged to fully memorize clean data in early training, and then discouraged with the increase of training epochs to reduce interference of mislabeled data. The updates of the parameters $\bmγ$ are the opposite. In testing, only the parameters $\bmσ$ are employed to enhance generalization. Extensive experiments on both simulated and real-world benchmarks confirm the superior performance of our method.

Figures (6)

• Figure 1: A toy example to illustrate the superiority of our method over the early stopping from the perspective of parameter space. The parameters of the origin network (shown as a blue circle) are decomposed into the sum of the ideal parameters (shown as a yellow circle) and the noise parameters (shown as a light blue circle). By making noise parameters absorb the partial effect of mislabeled data, and with the principle of parameter decomposition, this method can make the best ideal parameters (shown as a red triangle) closer to the optimal point of clean data, compared with the parameters trained by vanilla early stopping (shown as a blue triangle).
• Figure 2: Illustration of the values of $\beta_1$ and $\beta_2$ with varied epochs.
• Figure 3: Illustrations of justifying our method from the perspective of parameter space.
• Figure 4: Illustrations of the effect of different $\beta_2$ functions on F-MNIST and CIFAR-10.
• Figure 5: Illustrations of relationship between the hyper-parameter $c_2$ and the noise rate.