Depth Creates No Bad Local Minima
Haihao Lu, Kenji Kawaguchi
TL;DR
This work investigates whether depth, in the absence of nonlinearity, can introduce bad local minima in deep linear networks. It shows that while depth yields a non-convex loss surface, it does not create new bad local minima by relating the deep parameterization to a rank-constrained shallow model and proving equivalence of local minima under full-row-rank conditions. The main contributions are a pair of results: (i) every local minimum of the deep problem maps to a local minimum of a shallow, rank-constrained problem, and (ii) if the input is sufficiently informative, all local minima of the shallow problem are global, implying all local minima of the deep network are global. The approach provides a simpler, more general framework that extends beyond Frobenius loss and has potential implications for matrix completion and broader loss functions.
Abstract
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.