Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correction

Hongru Duan, Yongle Chen, Lei Guan

TL;DR

X-SAM addresses a gap in Sharpness-Aware Minimization by revealing that gradient updates can align unfavorably with the Hessian's dominant direction, limiting reduction of the maximum curvature. It introduces a gradient decomposition strategy that attenuates the component along the leading eigenvector, accompanied by intermittent eigenvector estimation and a correction term, yielding provable convergence and reduced top eigenvalue. Empirically, X-SAM consistently improves generalization over SAM and variants across CIFAR-10/100 and Fashion-MNIST, while producing a flatter Hessian spectrum with smaller $\lambda_1$ and trace. The approach offers a practical path to stronger robustness and generalization in deep nets, with future work focusing on reducing eigenvector estimation overhead and scaling to larger models.

Abstract

Sharpness-Aware Minimization (SAM) aims to improve generalization by minimizing a worst-case perturbed loss over a small neighborhood of model parameters. However, during training, its optimization behavior does not always align with theoretical expectations, since both sharp and flat regions may yield a small perturbed loss. In such cases, the gradient may still point toward sharp regions, failing to achieve the intended effect of SAM. To address this issue, we investigate SAM from a spectral and geometric perspective: specifically, we utilize the angle between the gradient and the leading eigenvector of the Hessian as a measure of sharpness. Our analysis illustrates that when this angle is less than or equal to ninety degrees, the effect of SAM's sharpness regularization can be weakened. Furthermore, we propose an explicit eigenvector-aligned SAM (X-SAM), which corrects the gradient via orthogonal decomposition along the top eigenvector, enabling more direct and efficient regularization of the Hessian's maximum eigenvalue. We prove X-SAM's convergence and superior generalization, with extensive experimental evaluations confirming both theoretical and practical advantages.

X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correction

TL;DR

X-SAM addresses a gap in Sharpness-Aware Minimization by revealing that gradient updates can align unfavorably with the Hessian's dominant direction, limiting reduction of the maximum curvature. It introduces a gradient decomposition strategy that attenuates the component along the leading eigenvector, accompanied by intermittent eigenvector estimation and a correction term, yielding provable convergence and reduced top eigenvalue. Empirically, X-SAM consistently improves generalization over SAM and variants across CIFAR-10/100 and Fashion-MNIST, while producing a flatter Hessian spectrum with smaller and trace. The approach offers a practical path to stronger robustness and generalization in deep nets, with future work focusing on reducing eigenvector estimation overhead and scaling to larger models.

Abstract

Sharpness-Aware Minimization (SAM) aims to improve generalization by minimizing a worst-case perturbed loss over a small neighborhood of model parameters. However, during training, its optimization behavior does not always align with theoretical expectations, since both sharp and flat regions may yield a small perturbed loss. In such cases, the gradient may still point toward sharp regions, failing to achieve the intended effect of SAM. To address this issue, we investigate SAM from a spectral and geometric perspective: specifically, we utilize the angle between the gradient and the leading eigenvector of the Hessian as a measure of sharpness. Our analysis illustrates that when this angle is less than or equal to ninety degrees, the effect of SAM's sharpness regularization can be weakened. Furthermore, we propose an explicit eigenvector-aligned SAM (X-SAM), which corrects the gradient via orthogonal decomposition along the top eigenvector, enabling more direct and efficient regularization of the Hessian's maximum eigenvalue. We prove X-SAM's convergence and superior generalization, with extensive experimental evaluations confirming both theoretical and practical advantages.
Paper Structure (18 sections, 3 theorems, 14 equations, 4 figures, 1 table)

This paper contains 18 sections, 3 theorems, 14 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Sharpness-Aware Minimization
  5. Methodology
  6. Empirical Analysis
  7. Gradient Decomposition
  8. The X-SAM Algorithm
  9. Theoretical Analysis
  10. Convergence Analysis
  11. Eigenvalue Decrease Enhances Generalization
  12. Orthogonality and Sharpness Reduction
  13. Experiments
  14. Experimental Settings
  15. Performance Comparisons
  16. ...and 3 more sections

Key Result

Theorem 5.1

Let $f:\mathbb{R}^d\to\mathbb{R}$ be $\beta$-smooth (i.e., $\nabla f$ is $\beta$-Lipschitz). Fix $\alpha\in[0,2]$ and assume $\|v_t\|=1$ for all $t$, and define the projector $P_t:=I-\alpha v_tv_t^\top , \text{so that} \|P_t\|_{\mathrm{op}}\le 1.$ Let $\rho>0$ and define $g_t:=\nabla f(x_t), h_t:=\n For non-convex stochastic optimization, Theorem D.2 shows that X-SAM achieves a convergence rate of

Figures (4)

  • Figure 1: Alignment statistics for a 6-layer CNN trained on CIFAR-10. The plots show the quantity distribution of the angle between the gradient (obtained from the second forward–backward pass of SAM) and the top Hessian eigenvector, which reflects the degree of alignment during training.
  • Figure 2: Experimental results when training ResNet-18 on CIFAR-10, CIFAR-100. Figs \ref{['fig:CIFAR-10-loss']}, \ref{['fig:CIFAR-10-acc']}: Test loss vs. Epochs; Figs \ref{['fig:CIFAR-100-loss']}, \ref{['fig:CIFAR-100-acc']}: Test accuracy vs. Epochs. Figs \ref{['fig:Fashion-MNIST-loss']}, \ref{['fig:Fashion-MNIST-acc']}: Top eigenvalue vs. Epochs.
  • Figure 3: Hessian spectrum at the end of training ResNet-18 on CIFAR-100.
  • Figure 4: Test accuracy of ResNet-18 when training on CIFAR-10 under different values of $\alpha$ for X-SAM.

Theorems & Definitions (3)

  • Theorem 5.1
  • Theorem 5.2: decrease of the leading eigenvalue
  • Theorem 5.3: first-order change of the leading eigenvalue