Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM

Hoang-Chau Luong, Thuc Nguyen-Quang, Minh-Triet Tran

TL;DR

This work proposes SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate, and demonstrates that SANER consistently outperforms SAM.

Abstract

Noisy labels pose a substantial challenge in machine learning, often resulting in overfitting and poor generalization. Sharpness-Aware Minimization (SAM), as demonstrated in Foret et al. (2021), improves generalization over traditional Stochastic Gradient Descent (SGD) in classification tasks with noisy labels by implicitly slowing noisy learning. While SAM's ability to generalize in noisy environments has been studied in several simplified settings, its full potential in more realistic training settings remains underexplored. In this work, we analyze SAM's behavior at each iteration, identifying specific components of the gradient vector that contribute significantly to its robustness against noisy labels. Based on these insights, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate. Our experiments on CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM, achieving up to an 8% increase on CIFAR-100 with 50% label noise.

Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM

TL;DR

This work proposes SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate, and demonstrates that SANER consistently outperforms SAM.

Abstract

Noisy labels pose a substantial challenge in machine learning, often resulting in overfitting and poor generalization. Sharpness-Aware Minimization (SAM), as demonstrated in Foret et al. (2021), improves generalization over traditional Stochastic Gradient Descent (SGD) in classification tasks with noisy labels by implicitly slowing noisy learning. While SAM's ability to generalize in noisy environments has been studied in several simplified settings, its full potential in more realistic training settings remains underexplored. In this work, we analyze SAM's behavior at each iteration, identifying specific components of the gradient vector that contribute significantly to its robustness against noisy labels. Based on these insights, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate. Our experiments on CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM, achieving up to an 8% increase on CIFAR-100 with 50% label noise.

Paper Structure

This paper contains 47 sections, 12 equations, 17 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Sharpness-Aware Minimization
  4. Related Works
  5. Analyzing Gradient Behavior of SAM
  6. Gradient weighting in SAM.
  7. Group B in SAM primarily mitigates noisy fitting.
  8. Analyzing the Impact of Group B Magnitude Reduction on slowing down Noisy vs. Clean Label Fitting
  9. Reweighting Group B for enhancing noisy label fitting resistance
  10. Sharpness-Aware Noise-Explicit Reweighting
  11. Value of $\alpha$ is directly proportional to noisy fitting rate.
  12. Stabilizing clean fitting via a scheduler for $\alpha$.
  13. Setup and Experimental Results
  14. Dataset.
  15. CIFAR-10 and CIFAR-100.
  16. ...and 32 more sections

Figures (17)

  • Figure 1: Performance comparison of SAM, SGD, and SANER (ours) trained on ResNet18 with CIFAR-10 under 25% label noise. Noise accuracy indicates how well the model overfits to noisy examples. SAM demonstrates the ability to slow down noisy fitting and increase the gap between clean and noisy accuracy, and our method can further enhance this effect. As a result, SANER outperforms SAM in test accuracy.
  • Figure 2: Parameter distribution (%) of groups A, B, and C during training.
  • Figure 3: Comparison of the noisy accuracy of SGD, SAM, and SAM variants where gradient components from groups A and B are swapped with those from SGD.
  • Figure 4: $pr$ value during training, showing that Group B has a greater influence on the noisy fitting rate.
  • Figure 5: Effect of hyperparameter $\alpha$ on noisy accuracy in (a) and (b). Lower values of $\alpha$ enhance noise resistance. In (c), compare clean accuracy of SANER with and without the $\alpha$ scheduler, demonstrating that the scheduler improves clean training accuracy.
  • ...and 12 more figures