Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guarantees
Yiming Zhang, Chester Holtz, Gal Mishne, Alex Cloninger
TL;DR
This work tackles learning with label noise by rigorously analyzing meta-reweighting methods that use a small trusted subset to guide training. It reveals a three-phase training dynamics where early alignment boosts clean examples, a filtering phase drives noisy weights to zero, and a post-filtering phase reduces discriminatory power as the clean-subset signal wanes. Guided by theory, the authors propose a lightweight surrogate, Feature-Based Reweighting (FBR), which preserves the essential signed, similarity-weighted meta-signal while replacing the costly bilevel updates with a mean-centered Gram-based computation. Empirically, the method yields consistent improvements over strong baselines on synthetic and real noisy-label benchmarks, showcasing scalable robustness and stability. The findings provide both theoretical insight into meta-reweighting under label noise and a practical, efficient algorithm that enhances generalization in the presence of noisy supervision.
Abstract
Learning with noisy labels remains challenging because over-parameterized networks memorize corrupted supervision. Meta-learning-based sample reweighting mitigates this by using a small clean subset to guide training, yet its behavior and training dynamics lack theoretical understanding. We provide a rigorous theoretical analysis of meta-reweighting under label noise and show that its training trajectory unfolds in three phases: (i) an alignment phase that amplifies examples consistent with a clean subset and suppresses conflicting ones; (ii) a filtering phase driving noisy example weights toward zero until the clean subset loss plateaus; and (iii) a post-filtering phase in which noise filtration becomes perturbation-sensitive. The mechanism is a similarity-weighted coupling between training and clean subset signals together with clean subset training loss contraction; in the post-filtering regime where the clean-subset loss is sufficiently small, the coupling term vanishes and meta-reweighting loses discriminatory power. Guided by this analysis, we propose a lightweight surrogate for meta-reweighting that integrates mean-centering, row shifting, and label-signed modulation, yielding more stable performance while avoiding expensive bi-level optimization. Across synthetic and real noisy-label benchmarks, our method consistently outperforms strong reweighting/selection baselines.