Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NCSAM Noise-Compensated Sharpness-Aware Minimization for Noisy Label Learning

Jiayu Xu, Junbiao Pang

TL;DR

This work addresses learning from noisy labels by revealing that label noise distorts SAM perturbations, hindering flat-minima optimization. It develops a PAC-Bayes–based theoretical framework to explain this distortion and introduces Noise-Compensated Sharpness-Aware Minimization (NCSAM), which combines progressive noise-label simulation with a dynamic compensation term to align perturbations with the clean-gradient direction. The method is supported by extensive experiments on synthetic and real-world noisy datasets, showing substantial robustness gains over SAM and competitive baselines, especially at higher noise levels. Importantly, NCSAM operates without explicit label correction or sample filtering and can augment existing LNL frameworks, offering a principled path toward robust perturbation-based optimization under weak supervision.

Abstract

Learning from Noisy Labels (LNL) presents a fundamental challenge in deep learning, as real-world datasets often contain erroneous or corrupted annotations, \textit{e.g.}, data crawled from Web. Current research focuses on sophisticated label correction mechanisms. In contrast, this paper adopts a novel perspective by establishing a theoretical analysis the relationship between flatness of the loss landscape and the presence of label noise. In this paper, we theoretically demonstrate that carefully simulated label noise synergistically enhances both the generalization performance and robustness of label noises. Consequently, we propose Noise-Compensated Sharpness-aware Minimization (NCSAM) to leverage the perturbation of Sharpness-Aware Minimization (SAM) to remedy the damage of label noises. Our analysis reveals that the testing accuracy exhibits a similar behavior that has been observed on the noise-clear dataset. Extensive experimental results on multiple benchmark datasets demonstrate the consistent superiority of the proposed method over existing state-of-the-art approaches on diverse tasks.

NCSAM Noise-Compensated Sharpness-Aware Minimization for Noisy Label Learning

TL;DR

This work addresses learning from noisy labels by revealing that label noise distorts SAM perturbations, hindering flat-minima optimization. It develops a PAC-Bayes–based theoretical framework to explain this distortion and introduces Noise-Compensated Sharpness-Aware Minimization (NCSAM), which combines progressive noise-label simulation with a dynamic compensation term to align perturbations with the clean-gradient direction. The method is supported by extensive experiments on synthetic and real-world noisy datasets, showing substantial robustness gains over SAM and competitive baselines, especially at higher noise levels. Importantly, NCSAM operates without explicit label correction or sample filtering and can augment existing LNL frameworks, offering a principled path toward robust perturbation-based optimization under weak supervision.

Abstract

Learning from Noisy Labels (LNL) presents a fundamental challenge in deep learning, as real-world datasets often contain erroneous or corrupted annotations, \textit{e.g.}, data crawled from Web. Current research focuses on sophisticated label correction mechanisms. In contrast, this paper adopts a novel perspective by establishing a theoretical analysis the relationship between flatness of the loss landscape and the presence of label noise. In this paper, we theoretically demonstrate that carefully simulated label noise synergistically enhances both the generalization performance and robustness of label noises. Consequently, we propose Noise-Compensated Sharpness-aware Minimization (NCSAM) to leverage the perturbation of Sharpness-Aware Minimization (SAM) to remedy the damage of label noises. Our analysis reveals that the testing accuracy exhibits a similar behavior that has been observed on the noise-clear dataset. Extensive experimental results on multiple benchmark datasets demonstrate the consistent superiority of the proposed method over existing state-of-the-art approaches on diverse tasks.
Paper Structure (35 sections, 29 equations, 1 figure, 7 tables, 1 algorithm)

This paper contains 35 sections, 29 equations, 1 figure, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning from Noisy Labels
  4. Sharpness and Noisy Labels
  5. Method
  6. Problem setup
  7. Theoretical Motivation
  8. PAC-Bayes–Motivated Interpretation of Perturbation Distortion under Noisy Labels
  9. PAC-Bayes Preliminaries.
  10. Noise-Induced Posterior Shift.
  11. SAM Perturbation as Posterior Smoothing.
  12. Implications for Noisy-Label Learning.
  13. Remark (Additive Interaction Between Noise and Perturbation).
  14. Distortion of SAM Perturbations under Noisy Labels
  15. SAM Perturbation under Clean Labels.
  16. ...and 20 more sections

Figures (1)

  • Figure 1: The accuracy curves of ResNet-18 trained on CIFAR-10 with 40% random noise labels using three optimization methods (NCSAM, SAM, and SGD) are shown as a function of training epochs.