Small nonlinearities in activation functions create bad local minima in neural networks

TL;DR

The paper investigates the loss surfaces of neural networks and shows that spurious local minima are common once nonlinearity is introduced, even for very simple architectures. It provides a constructive spurious local minimum for ReLU-like activations and a broader counterexample for a wide range of activations on realizable data, while also delivering a unifying global-optimality result for deep linear networks. A key contribution is tying the nonlinear and linear cases with a multilinear-parametrization framework, clarifying when local minima are global minima or saddles. These insights imply that the favorable property of no spurious local minima, often associated with linear networks, does not robustly extend to nonlinear networks, shaping our understanding of optimization landscapes in practice.

Abstract

We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.

Theorems & Definitions (11)

• Theorem 1
• Theorem 2
• Corollary 3
• Theorem 4
• Corollary 5
• Lemma A.1
• Lemma A.2
• Lemma A.3
• Lemma A.4
• Lemma A.5