Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
Rong Ge, Furong Huang, Chi Jin, Yang Yuan
TL;DR
<3-5 sentence high-level summary> This work studies stochastic gradient descent in non-convex optimization, where saddle points impede progress. It identifies a robust strict-saddle property and proves that SGD with noise can escape saddles and converge to a local minimum in poly-time, even when exponentially many local minima and saddles exist. The authors apply this framework to online orthogonal tensor decomposition by formulating a strict-saddle objective whose minima correspond to the true components, yielding the first online algorithm with global convergence guarantees. The results have implications for scalable learning in latent-variable models and other symmetry-rich non-convex problems, providing a principled path toward reliable training in settings where second-order methods are impractical.</p>
Abstract
We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. Our analysis can be applied to orthogonal tensor decomposition, which is widely used in learning a rich class of latent variable models. We propose a new optimization formulation for the tensor decomposition problem that has strict saddle property. As a result we get the first online algorithm for orthogonal tensor decomposition with global convergence guarantee.