GCSAM: Gradient Centralized Sharpness Aware Minimization

TL;DR

This work tackles the generalization gap in overparameterized DNNs by addressing sharpness in the loss landscape. It introduces Gradient-Centralized Sharpness Aware Minimization (GCSAM), which blends Gradient Centralization with SAM by centering gradients in the ascent step, using perturbations $oldsymbol{\epsilon}_{GC}=\rho \frac{\nabla_{GC} L_S(\boldsymbol{w})}{\|\nabla_{GC} L_S(\boldsymbol{w})\|_p}$ and optimizing $\min_{\boldsymbol{w}} \max_{\|\epsilon_{GC}\|_p\le \rho} L_S(\boldsymbol{w}+\epsilon_{GC}) + \frac{\lambda}{2}\frac{\|\boldsymbol{w}\|_2^2}{\rho^2}$. Empirically, GCSAM outperforms Adam and SAM variants on CIFAR-10 and medical imaging tasks (BUS and COVID-19), with better generalization and faster convergence, attributed to reduced gradient noise and tighter sharpness bounds. The approach yields consistently flatter loss landscapes and robust performance across CNNs and Vision Transformers, suggesting practical value for domains with data variability and domain shifts. Future work may explore momentum-based variants and adaptive hyperparameters to further enhance efficiency and scalability.

Abstract

The generalization performance of deep neural networks (DNNs) is a critical factor in achieving robust model behavior on unseen data. Recent studies have highlighted the importance of sharpness-based measures in promoting generalization by encouraging convergence to flatter minima. Among these approaches, Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape, thereby improving generalization. However, SAM's computational overhead and sensitivity to noisy gradients limit its scalability and efficiency. To address these challenges, we propose Gradient-Centralized Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize gradients and accelerate convergence. GCSAM normalizes gradients before the ascent step, reducing noise and variance, and improving stability during training. Our evaluations indicate that GCSAM consistently outperforms SAM and the Adam optimizer in terms of generalization and computational efficiency. These findings demonstrate GCSAM's effectiveness across diverse domains, including general and medical imaging tasks.

Figures (3)

• Figure 1: (a) Sketch map for using Gradient Centralization (GC). $L$ is the loss function, $\textbf{w}$ denotes the weight matrix, $\nabla L_S(\bold{w})$ is the gradient of the entire weight matrix, and $\nabla_{GC} L_S(\bold{w})$ is the centralized gradient. (b) Geometrical interpretation of GC. The gradient is projected on a hyperplane $\hat{\mu}$, where the projected gradient is used to update the weight.
• Figure 2: (a) GCSAM Algorithm. (b) Schematic of the GCSAM paramter update, where $\textbf{w}_t$ is the current step, $\epsilon_{GC}$ perturbs weights fot the ascent step, and $\textbf{w}_{t+1}$ is the next step.
• Figure 3: Comparison of loss landscapes for Adam, SAM, and GCSAM on the COVID-19 dataset. Adam demonstrates the sharpest minima, reflecting less stable convergence. In contrast, GCSAM achieves the flattest minima, indicative of improved stability and better convergence properties.

Theorems & Definitions (2)

• Theorem A.1
• proof