How to Escape Saddle Points Efficiently
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, Michael I. Jordan
TL;DR
This work introduces Perturbed Gradient Descent (PGD) for non-convex optimization and proves that it converges to an $ extepsilon$-second-order stationary point in nearly dimension-free time, up to polylog factors. The analysis hinges on a novel saddle-point geometry showing a thin stuck region, enabling random perturbations to effectively escape saddles. Under the strict saddle property, PGD guarantees convergence to local minima, and with local regularity or strong local structure, it achieves linear convergence via PGD plus a local-improvement phase. The matrix factorization example demonstrates sharp global convergence rates, illustrating the practical impact for non-convex learning problems. Overall, the results provide almost dimension-free guarantees for escaping saddles and finding local minima in broad non-convex settings.
Abstract
This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i.e., it is almost "dimension-free"). The convergence rate of this procedure matches the well-known convergence rate of gradient descent to first-order stationary points, up to log factors. When all saddle points are non-degenerate, all second-order stationary points are local minima, and our result thus shows that perturbed gradient descent can escape saddle points almost for free. Our results can be directly applied to many machine learning applications, including deep learning. As a particular concrete example of such an application, we show that our results can be used directly to establish sharp global convergence rates for matrix factorization. Our results rely on a novel characterization of the geometry around saddle points, which may be of independent interest to the non-convex optimization community.