First-order Methods Almost Always Avoid Saddle Points
Jason D. Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I. Jordan, Benjamin Recht
TL;DR
The paper addresses the prevalence of saddle points in non-convex optimization and shows that a wide class of first-order methods almost surely avoid saddle points when initialized randomly. By treating optimization updates as dynamical systems and applying the Stable Manifold Theorem, it proves that the basins of attraction for saddle points have measure zero, under mild smoothness and invertibility conditions. The results cover gradient descent, proximal point, coordinate and block coordinate descent, manifold gradient descent, and mirror descent, providing uniform criteria under which saddles are unstable fixed points. This deterministic analysis offers a unified foundation for why simple first-order methods typically converge to local minima and suggests directions for designing efficient, provably robust optimization procedures without relying on stochastic perturbations or second-order information.
Abstract
We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid saddle points.