Accelerated Gradient Methods for Nonconvex Optimization: Escape Trajectories From Strict Saddle Points and Convergence to Local Minima

TL;DR

This work studies a broad class of accelerated gradient methods applied to smooth nonconvex functions, focusing on escaping strict saddle points and converging to local minima. It develops an asymptotic framework using almost-sure diffeomorphism properties and Jacobian-eigenvalue analysis to show almost-sure avoidance of strict saddles for time-varying momentum schemes, and introduces two metrics that quantify asymptotic convergence/divergence rates near critical points. It then provides non-asymptotic exit-time results (linear in iterations) for trajectories near weakly hyperbolic fixed points, including a detailed complex-dynamics analysis and a subsidiary sub-class that achieves near-optimal local convergence in convex neighborhoods while preserving superior saddle-escape behavior. The paper also furnishes yields for concrete algorithms (Nesterov variants, fixed and variable momentum) and demonstrates, via numerical experiments on phase retrieval, low-rank matrix factorization, and positive definite quadratic problems, that larger limiting momentum can accelerate saddle escape, with a tradeoff in local convergence speed. Overall, the results equip practitioners with principled guidance on momentum choices to balance fast saddle escape against robust local convergence, and advance the theoretical understanding of accelerated methods in nonconvex optimization.

Abstract

This paper considers the problem of understanding the behavior of a general class of accelerated gradient methods on smooth nonconvex functions. Motivated by some recent works that have proposed effective algorithms, based on Polyak's heavy ball method and the Nesterov accelerated gradient method, to achieve convergence to a local minimum of nonconvex functions, this work proposes a broad class of Nesterov-type accelerated methods and puts forth a rigorous study of these methods encompassing the escape from saddle points and convergence to local minima through both an asymptotic and a non-asymptotic analysis. In the asymptotic regime, this paper answers an open question of whether Nesterov's accelerated gradient method (NAG) with variable momentum parameter avoids strict saddle points almost surely. This work also develops two metrics of asymptotic rates of convergence and divergence, and evaluates these two metrics for several popular standard accelerated methods such as the NAG and Nesterov's accelerated gradient with constant momentum (NCM) near strict saddle points. In the non-asymptotic regime, this work provides an analysis that leads to the "linear" exit time estimates from strict saddle neighborhoods for trajectories of these accelerated methods as well the necessary conditions for the existence of such trajectories. Finally, this work studies a sub-class of accelerated methods that can converge in convex neighborhoods of nonconvex functions with a near optimal rate to a local minimum and at the same time this sub-class offers superior saddle-escape behavior compared to that of NAG.

Figures (9)

• Figure 1: Heatmaps of (a) the lower bound on the asymptotic divergence metric $\mathcal{M}^{\star}(f)$ from Theorem \ref{['metricedivergethm']} and (b) the upper bound on the exit time metric $K_{\text{exit}}(\sigma)$ from \ref{['exittimetradeoffbound']}, shown as functions of the asymptotic momentum parameter $\beta$ and step size $h$. While the scales differ across the two plots, the bounds exhibit an order-wise inverse relationship: as $\beta$ and $h$ increase, the lower bound on $\mathcal{M}^{\star}(f)$ increases, whereas the upper bound on $K_{\text{exit}}(\sigma)$ decreases.
• Figure 2: Simulated trajectories of various accelerated gradient methods from the family \ref{['familyof momentum']} on the phase retrieval problem with $h = 0.01/L$, under the same initial unstable projections for fixed values of $m$, $n$, and $\epsilon$. The parameter $r$ controls the momentum via $\beta_k = \frac{k}{k + 3 - r}$; $r = 0$ corresponds to the Nesterov accelerated method \ref{['originalnesterov']}.
• Figure 3: Simulated trajectories of various accelerated gradient methods from the family \ref{['familyof momentum']} on the phase retrieval problem with $h = 0.1/L$, under the same initial unstable projections for fixed values of $m$, $n$, and $\epsilon$. The parameter $r$ controls the momentum via $\beta_k = \frac{k}{k + 3 - r}$; $r = 0$ corresponds to the Nesterov accelerated method \ref{['originalnesterov']}.
• Figure 4: Simulated trajectories of various accelerated gradient methods from \ref{['ds1']} with $\beta_k = \frac{r k}{k + 3 - r}$ on the phase retrieval problem with $h = 0.01/L$, under the same initial unstable projections for fixed values of $m$, $n$, and $\epsilon$. Note that $r = 0$ corresponds to the gradient descent method.
• Figure 5: Simulated trajectories of various accelerated gradient methods from \ref{['ds1']} with $\beta_k = \frac{r k}{k + 3 - r}$ on the phase retrieval problem with $h = 0.01/L$, under different initial unstable projections for fixed values of $m$, $n$, and $\epsilon$. Note that $r = 0$ corresponds to the gradient descent method.

Theorems & Definitions (100)

• Theorem 3.1
• Corollary 3.2
• Remark 3.3
• Corollary 3.4
• Theorem 3.5
• Lemma 3.6
• Corollary 3.7
• Theorem 3.8
• Theorem 3.9
• Theorem 3.10