Statistical Guarantees for Approximate Stationary Points of Shallow Neural Networks

TL;DR

The paper tackles the gap between theory and practice in neural networks by proving statistical guarantees for stationary points of shallow networks (linear and ReLU). It develops rigor around regularized least-squares objectives, showing that any reasonable stationary point generalizes nearly as well as the target in expectation, with rates matching global optima up to log factors. The results extend to ReLU networks under a first-layer orthogonality-type assumption and identify optimal tuning parameters for regularization across noise regimes, including heavy-tailed settings. Numerical experiments corroborate that approximate stationary points can achieve test performance close to that of global optima, supporting the pragmatic view that exact global optimization is not always necessary in deep learning.

Abstract

Since statistical guarantees for neural networks are usually restricted to global optima of intricate objective functions, it is unclear whether these theories explain the performances of actual outputs of neural network pipelines. The goal of this paper is, therefore, to bring statistical theory closer to practice. We develop statistical guarantees for shallow linear neural networks that coincide up to logarithmic factors with the global optima but apply to stationary points and the points nearby. These results support the common notion that neural networks do not necessarily need to be optimized globally from a mathematical perspective. We then extend our statistical guarantees to shallow ReLU neural networks, assuming the first layer weight matrices are nearly identical for the stationary network and the target. More generally, despite being limited to shallow neural networks for now, our theories make an important step forward in describing the practical properties of neural networks in mathematical terms.

Figures (4)

• Figure 1: Since objective functions in deep learning are usually highly non-convex and cannot be solved explicitly, we can only expect approximate stationary points from practical algorithms.
• Figure 2: Non-convex objective function $\min_{(a_1,a_2)} f_{(a_1,a_2)}(X)=\sum_{i=1}^{n}(a_1\sigma(a_2x_i)-y_i)^2/2+|a_1|+|a_2|$ for two training samples $(x_1=2,y_1=2)$ and $(x_2=4,y_2=1)$ includes critical points that are not global optima. The left panel illustrates the objective for linear activation function, and the right panel shows the objective for the ReLU.
• Figure 3: Log-training error for neural networks (with $\textcolor{black}{$d$}=\textcolor{black}{$w$}=10$) with linear (left panel) and ReLU (right panel) activations in $10$ different runs (allocated with different colors). Due to the non-convexity of neural networks, optimization algorithms may end up in different approximate stationary points.
• Figure 4: Relative error versus tuning parameter for shallow networks. Left: linear activation; right: ReLU activation. The results clearly illustrate a bias–variance trade-off when the tuning parameter is either too large or too small.

Theorems & Definitions (38)

• Theorem 1: Statistical Guarantees for Reasonable Stationary Points of Shallow Linear Networks
• Theorem 2: Statistical Guarantees for Approximate Stationary Points of Shallow Linear Networks
• Theorem 3: Statistical Guarantees for Reasonable Stationary Points of Shallow ReLU Networks
• Proposition 1: Hessian Behavior for Shallow Linear Network
• Lemma 1: Empirical Processes
• Proposition 2: Hessian Behavior for Shallow ReLU Networks
• Remark 1: Empirical Processes for Shallow ReLU Neural Networks
• Definition 1: Tails
• Theorem 4: Statistical Guarantees for Reasonable Stationary Points for Heavy-tailed Noise
• Example 1: Existence of sub-optimal critical points for regularized shallow networks