Coordinated Sparse Recovery of Label Noise

TL;DR

Label noise, especially instance-dependent, degrades generalization in deep models. The authors propose Coordinated Sparse Recovery (CSR), introducing a learnable collaboration matrix $M$ and confidence weights to align model updates with sparse noise recovery, and analyze why SOP suffers from gradient leakage due to misaligned learning paces. Building on CSR, they develop CSR+ with joint sample selection, consistency regularization, Mixup, and dynamic noisy-label correction to further suppress confirmation bias and improve robustness. Across synthetic and real noisy datasets, CSR and CSR+ achieve strong performance, particularly as the number of classes grows and noise becomes highly instance-specific, indicating practical impact for robust learning under noisy labels.

Abstract

Label noise is a common issue in real-world datasets that inevitably impacts the generalization of models. This study focuses on robust classification tasks where the label noise is instance-dependent. Estimating the transition matrix accurately in this task is challenging, and methods based on sample selection often exhibit confirmation bias to varying degrees. Sparse over-parameterized training (SOP) has been theoretically effective in estimating and recovering label noise, offering a novel solution for noise-label learning. However, this study empirically observes and verifies a technical flaw of SOP: the lack of coordination between model predictions and noise recovery leads to increased generalization error. To address this, we propose a method called Coordinated Sparse Recovery (CSR). CSR introduces a collaboration matrix and confidence weights to coordinate model predictions and noise recovery, reducing error leakage. Based on CSR, this study designs a joint sample selection strategy and constructs a comprehensive and powerful learning framework called CSR+. CSR+ significantly reduces confirmation bias, especially for datasets with more classes and a high proportion of instance-specific noise. Experimental results on simulated and real-world noisy datasets demonstrate that both CSR and CSR+ achieve outstanding performance compared to methods at the same level.

Figures (9)

• Figure 1: Empirical analysis of the incoordination phenomenon of $v$-lag learning and its impact on test error on CIFAR-10/100 Datasets with different label noises: variation trends of parameter gradient with respect to training time (a)$\sim$(h) and correlation between parameter learning incoordination and model generalization (i)$\sim$(l).
• Figure 2: Visualization and change analysis of M matrix values: (a) average value variations in M matrix over time; (b) visualization of M value at 50 epoch time: mainly concentrated on the diagonal
• Figure 3: Comparison on an incorrectly annotated sample (b) with and (a) without the use of a collaboration matrix. Without the collaboration matrix, there would be an error loss of 0.69, whereas, after the utilization of the collaboration matrix, the error loss decreases significantly to 0.18.
• Figure 4: The architecture of CSR.
• Figure 5: Overall Algorithm of CSR.