The Speed-Robustness Trade-Off for First-Order Methods with Additive Gradient Noise

TL;DR

This work tackles the speed-robustness trade-off in first-order optimization under additive gradient noise. It introduces a fixed fixed-parameter framework with a three-parameter update rule $x^{t+1}=x^t - \alpha g\bigl(x^t+\eta(x^t-x^{t-1})\bigr) + \beta(x^t-x^{t-1})$ and develops Lyapunov-LMI certificates to bound both convergence rate and noise sensitivity for two function classes, $Q_{m,L}$ and $F_{m,L}$. The authors present two algorithms, Robust Heavy Ball (RHB) for quadratics and Robust Accelerated Method (RAM) for smooth strongly convex functions, providing explicit analytic parameterizations in terms of problem constants and a tunable scalar $r$ that directly trades off rate and robustness. Numerical studies and worst-case simulations show RAM and RHB achieve Pareto-optimal rate-sensitivity trade-offs, outperforming standard methods in these settings and offering a practical tool for navigating speed versus robustness in stochastic optimization with fixed stepsizes.

Abstract

We study the trade-off between convergence rate and sensitivity to stochastic additive gradient noise for first-order optimization methods. Ordinary Gradient Descent (GD) can be made fast-and-sensitive or slow-and-robust by increasing or decreasing the stepsize, respectively. However, it is not clear how such a trade-off can be navigated when working with accelerated methods such as Polyak's Heavy Ball (HB) or Nesterov's Fast Gradient (FG) methods. We consider two classes of functions: (1) strongly convex quadratics and (2) smooth strongly convex functions. For each function class, we present a tractable way to compute the convergence rate and sensitivity to additive gradient noise for a broad family of first-order methods, and we present algorithm designs that trade off these competing performance metrics. Each design consists of a simple analytic update rule with two states of memory, similar to HB and FG. Moreover, each design has a scalar tuning parameter that explicitly trades off convergence rate and sensitivity to additive gradient noise. We numerically validate the performance of our designs by comparing their convergence rate and sensitivity to those of many other algorithms, and through simulations on Nesterov's "bad function".

Figures (5)

• Figure 1: Trade-off between convergence rate and steady-state error (sensitivity to noise). Three different tunings of Gradient Descent (GD) with additive gradient noise are applied to random strongly convex quadratic functions on $\mathbb{R}^{10}$. Half of the Hessian eigenvalues are at $m=1$, the other half at $L=10$. The initialization is $x^0 = 1000\,\mathsf{e}_1$. Gradient noise is normally distributed $\mathcal{N}(0,I)$ and i.i.d. across iterations. The plot shows mean and $\pm 1$ standard deviation of the error $\lVert{x^t-x^\star}\rVert$ for 1000 sample trajectories. On the right panel, iterations are plotted on a log scale.
• Figure 2: Plot of sensitivity squared vs. rate squared for algorithms applied to the function class $Q_{1,10}$ (left panel) and $Q_{1,100}$ (right panel), found using \ref{['cor:perf_quad']}. Each point in the point cloud corresponds to an algorithm $(\alpha,\beta,\eta)$. The Pareto-optimal front coincides with our proposed Robust Heavy Ball method (RHB, \ref{['thm:RHB']}), tuned using $r \in \bigl[ \frac{\sqrt{L}-\sqrt{m}}{\sqrt{L}+\sqrt{m}}, 1\bigr]$ to mediate the trade-off.
• Figure 3: Plot of squared sensitivity vs. squared rate for algorithms applied to the function class $F_{1,10}$ (left panel) and $F_{1,100}$ (right panel), found using \ref{['thm:lmi_cvx']}. Each point in the point cloud corresponds to an algorithm $(\alpha,\beta,\eta)$. RM and GD are strictly suboptimal, and the Pareto-optimal front closely matches RAM (\ref{['thm:RAM']}), tuned using $r \in \bigl[ 1-\sqrt{\frac{m}{L}}, 1\bigr]$ to mediate the trade-off. For reference, the bound \ref{['eq:AGD']} on $\mu$AGD+ from cohen2018 is also shown.
• Figure 4: Left: Regions of the sensitivity vs. rate trade-off space for $F_{1,100}$ covered by the three-parameter family $(\alpha,\beta,\eta)$, the Nesterov (Fast Gradient) family $(\alpha,\beta,\beta)$, and the Polyak (Heavy Ball) family $(\alpha,\beta,0)$. The FG and HB families are not expressive enough to capture the whole trade-off space. Right: Comparison of RAM with the numerically synthesized algorithms (using a state dimension up to $n=6$) from scherer for $F_{1,50}$. RAM outperforms despite using only two states of memory. We plot the log of the normalized sensitivity vs. the rate to match scherer.
• Figure 5: Simulations of various algorithms with low noise ($\sigma=10^{-5}$, left column) and higher noise ($\sigma=10^{-2}$, right column). Each algorithm was simulated on Nesterov's lower-bound quadratic function, with $m=1$, $L=10$, and dimension $d=100$. Shaded regions indicate $\pm 1$ standard deviations about the mean across $100$ trials (different noise realizations). Different tunings of our proposed Robust Heavy Ball (RHB) from \ref{['thm:RHB']} effectively trade off convergence rate and steady-state error (sensitivity to noise). Bottom row: the red curve shows RHB with (hand-tuned) piecewise constant $r$, where the red dots indicate switch points.

Theorems & Definitions (22)

• Remark 1: important notational convention
• Remark 2
• Lemma 2.1: three-parameter family
• Proposition 2.2
• Remark 3
• Lemma 3.1: $Q_{m,L}$ analysis
• Corollary 3.2: $Q_{m,L}$ analysis, reduced
• Theorem 3.3: GD analysis for $Q_{m,L}$
• Theorem 3.4: Robust Heavy Ball, RHB
• Lemma 4.1: Lyapunov analysis for rate of convergence