Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

Ioannis Panageas, Georgios Piliouras

TL;DR

The paper addresses the threat of saddle points in non-convex optimization by showing that gradient descent converges to saddles only on a measure-zero set of initial conditions, even when critical points are non-isolated. It extends previous results by removing strict isolation requirements and, in forward-invariant domains, relaxing global Lipschitz assumptions while providing an explicit step-size bound. The authors combine diffeomorphism properties of the gradient map with center-stable manifold theory and measure-theoretic arguments, and illustrate the results with examples highlighting non-isolated points, forward-invariant regions, and step-size effects. The findings strengthen the understanding of gradient-descent robustness and offer practical guidance for step-size selection and domain choices in non-convex settings.

Abstract

Given a non-convex twice differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where \nabla^2 f has at least one strictly negative eigenvalue has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.

Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

TL;DR

The paper addresses the threat of saddle points in non-convex optimization by showing that gradient descent converges to saddles only on a measure-zero set of initial conditions, even when critical points are non-isolated. It extends previous results by removing strict isolation requirements and, in forward-invariant domains, relaxing global Lipschitz assumptions while providing an explicit step-size bound. The authors combine diffeomorphism properties of the gradient map with center-stable manifold theory and measure-theoretic arguments, and illustrate the results with examples highlighting non-isolated points, forward-invariant regions, and step-size effects. The findings strengthen the understanding of gradient-descent robustness and offer practical guidance for step-size selection and domain choices in non-convex settings.

Abstract

Given a non-convex twice differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where \nabla^2 f has at least one strictly negative eigenvalue has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.

Paper Structure

This paper contains 13 sections, 10 theorems, 10 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Organization
  4. Preliminaries
  5. Main Results
  6. Proving the theorems
  7. Proof of Theorem \ref{['thm:main2']}
  8. Proof of Theorem \ref{['thm:main3']}
  9. Examples
  10. Example for non-isolated critical points
  11. Example for forward invariant set
  12. Example for step-size
  13. Conclusion

Key Result

Theorem 2

[Non-isolated] Let $f: \mathbb{R}^N \to \mathbb{R}$ be a twice continuously differentiable function and $\sup_{\mathbf{x} \in \mathbb{R}^N} \left\Vert \nabla^2 f(\mathbf{x}) \right\Vert_{2} \leq L < \infty$. The set of initial conditions $\mathbf{x} \in \mathbb{R}^N$ so that gradient descent with st

Figures (2)

  • Figure 1: Example that satisfies the assumptions of Theorem \ref{['thm:main1']}. The black line represent critical points of $f$, all of which are strict. The red lines correspond to diverging trajectories of gradient descent with small step size.
  • Figure 2: Example that satisfies the assumptions of Theorem \ref{['thm:main2']}. The three black dots represent the critical points. Function $f$ is not Lipschitz.

Theorems & Definitions (19)

  • Definition 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 5
  • proof
  • Remark 6
  • Lemma 7
  • proof
  • Lemma 8
  • ...and 9 more