Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

Xuchen Gong, Tian Li

TL;DR

This work bridges zeroth-order optimization and sharpness-aware minimization by introducing ZEST, a gradient-free framework that uses exponential tilting to form a continuum of objectives between average and max loss. It derives unbiased zeroth-order gradient estimators for the tilted SAM objective, and develops ratio-of-expectations estimators with bias-corrected variants to enable practical updates with finite perturbations. Theoretical analysis characterizes the sharpness regularizer $R_t$ under Gaussian and ball perturbations, showing how larger $t$ concentrates penalties on large Hessian eigenvalues, promoting flatter minima. Empirically, ZEST demonstrates memory-efficient training and competitive or superior generalization relative to vanilla zeroth-order baselines and first-order SAM variants across classification, QA, and language-generation tasks. The results validate exponential tilting as an effective mechanism to blend average-case robustness with curvature-aware optimization in a gradient-free setting.

Abstract

Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average- and the max-loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter $t$. We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation.

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

TL;DR

This work bridges zeroth-order optimization and sharpness-aware minimization by introducing ZEST, a gradient-free framework that uses exponential tilting to form a continuum of objectives between average and max loss. It derives unbiased zeroth-order gradient estimators for the tilted SAM objective, and develops ratio-of-expectations estimators with bias-corrected variants to enable practical updates with finite perturbations. Theoretical analysis characterizes the sharpness regularizer under Gaussian and ball perturbations, showing how larger concentrates penalties on large Hessian eigenvalues, promoting flatter minima. Empirically, ZEST demonstrates memory-efficient training and competitive or superior generalization relative to vanilla zeroth-order baselines and first-order SAM variants across classification, QA, and language-generation tasks. The results validate exponential tilting as an effective mechanism to blend average-case robustness with curvature-aware optimization in a gradient-free setting.

Abstract

Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average- and the max-loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter . We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation.
Paper Structure (44 sections, 5 theorems, 88 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 44 sections, 5 theorems, 88 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Work
  3. Zeroth-Order Optimization.
  4. Sharpness-Aware Minimization.
  5. Zeroth-Order Tilted Sharpness-Aware Learning
  6. Tilted Zeroth-Order Gradient
  7. Estimates of Ratio-of-Expectations
  8. Naive Plug-In.
  9. Bias-Corrected Plug-In.
  10. Sharpness Notions
  11. Gaussian Perturbation
  12. Ball Perturbation
  13. Low-Dimensional Examples
  14. Experiments
  15. Baselines.
  16. ...and 29 more sections

Key Result

Theorem 3.1

Denote $\mathcal{N}\coloneqq \mathcal{N}(0, I_d)$, $\mathcal{S}\coloneqq \mathcal{U}(\sqrt{d}\mathbb{S}^{d-1})$, i.e., uniform distribution over the sphere $\{v\in \mathbb{R}^d: \left\lVert v\right\rVert=\sqrt{d}\}$, and $\mathcal{B}\coloneqq \mathcal{U}(\sqrt{d}\mathbb{B}^d)$, i.e., uniform distrib (2) with $F_t(x)=\frac{1}{t} \log \mathbb{E}_{v\sim\mathcal{B}} [e^{tf(x+\rho v)}]$, we have

Figures (4)

  • Figure 1: Convergence behaviors of different methods on examples for the (a) linear and (b) stationary regimes. It shows that (1) MeZO can make steep steps while ZEST identifies flat next iterations, and (2) MeZO can converge to minima with a large $\lambda_{\max}$ while ZEST explicitly biases against $\lambda_{\max}$.
  • Figure 2: Sharpness of the solutions found by MeZO and ZEST on MRPC. Left: Scatters denote the average loss of the neighborhood among 500 perturbations, and the shade denotes the standard deviation. Right: Top-5 eigenvalues of Hessian.
  • Figure 3: Validation accuracies of MeZO ($t=0$) and ZEST ($t=\{1,5,20\}$) on different datasets with clean labels (Upper) and 30% noisy labels (Bottom). The x-axis denotes evaluation steps. On each dataset, we have $k=5$ sampled perturbations. The curves are smoothed for visualization, so we report the final smoothed accuracy and the final raw accuracy in the brackets. The results show that $t=1$ almost always outperforms MeZO ($t=0$): In the above plots, $t=1$ outperforms $t=0$ in raw accuracies by 1.7%, 0.8%, 0.4%, 2.9%, 5.7%, 0.8% and underperform by only 0.1% on SST-5.
  • Figure 4: Sharpness of the solutions found by MeZO and ZEST on MNLI, SST-5, and RTE with RoBERTa-Base. Upper: Sharpness measured by $\mathbb{E}_{\left\lVert\epsilon\right\rVert\leq \rho}[f(x+\epsilon)]$. The scatters denote the average loss among 500 sampled perturbations and the shade denotes the standard deviation. Lower: Sharpness measured by the top-5 eigenvalues of $\nabla^2 f(x)$. The results suggest that ZEST yields flatter solutions in terms of both the robustness to parameter perturbations and largest curvature of the loss landscape at the arrived minimum, which agrees with our theoretical analysis in Section \ref{['sec:sharpness']}.

Theorems & Definitions (13)

  • Theorem 3.1: Tilted Zeroth-Order Gradient
  • Definition 4.1: Sharpness Sensitivity
  • Theorem 4.1: Sharpness under Gaussian Perturbation
  • Theorem 4.2: Sharpness under Ball Perturbation
  • proof
  • proof
  • Lemma B.1: Measure of Concentration
  • proof
  • proof
  • proof
  • ...and 3 more