Gradient Span Algorithms Make Predictable Progress in High Dimension

Abstract

We prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity. In particular, this result explains the counterintuitive phenomenon that different training runs of many large machine learning models result in approximately equal cost curves despite random initialization on a complicated non-convex landscape. The distributional assumption of (non-stationary) isotropic Gaussian random functions we use is sufficiently general to serve as realistic model for machine learning training but also encompass spin glasses and random quadratic functions.

Figures (1)

• Figure 1: The plot shows an empirical approximation of the cost sequence resulting from the training of a standard convolutional neural network on the MNIST dataset lecunMNISTDATABASEHandwritten2010. We plot the values of $f(x_0^{(i)}),\dots, f(x_{120}^{(i)})$ against the steps $0,\dots,120$ on the $x$-axis. The minimization is performed with three optimization algorithms: Adam kingmaAdamMethodStochastic2015 (with learning rate $\eta$ and momentum $\beta$) in blue and two version of gradient descent (learning rate $\eta=0.1$ and $\eta=3$) in red and green. Each optimizer was run $10$ times from randomly selected initializations $x_0^{(i)}$ using the (random) default initialization procedure known as Glorot initialization glorotUnderstandingDifficultyTraining2010 (further details in Appendix \ref{['appdx: ml']}).

Theorems & Definitions (83)

• Remark 1.2: Counterintuitive results
• Definition 2.1: General gradient span algorithm
• Remark 2.2: Projection
• Definition 2.3: Scaled sequence of isotropic Gaussian random functions (GRF)
• Remark 2.4: "Function"
• Remark 2.5: Alternative definition
• Remark 2.6: Scaling
• Lemma 2.7
• proof
• Definition 2.8: Valid in all dimensions