Global $\mathcal{L}^2$ minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning
Thomas Chen
TL;DR
This paper investigates gradient-descent dynamics in deep learning by exploiting the freedom to choose the underlying Riemannian metric, focusing on Euclidean gradient flow in the network output layer and two geometrically adapted flows for overparametrized and underparametrized regimes. It proves that, under a rank condition on the Jacobian $D$, the overparametrized modified flow achieves global exponential convergence of the $L^2$ loss to zero and yields an a priori stopping time; it also develops a framework for rank-loss cases leading to constrained/differential-algebraic dynamics and connects these flows to sub-Riemannian geometry. In the underparametrized regime, the modified flow maps to a constrained gradient flow in the output layer; the borderline case $K=QN$ yields a unification of the two approaches. These results illuminate the geometric structure underlying training dynamics and offer principled stopping criteria and insight into local equilibria.
Abstract
We consider the scenario of supervised learning in Deep Learning (DL) networks, and exploit the arbitrariness of choice in the Riemannian metric relative to which the gradient descent flow can be defined (a general fact of differential geometry). In the standard approach to DL, the gradient flow on the space of parameters (weights and biases) is defined with respect to the Euclidean metric. Here instead, we choose the gradient flow with respect to the Euclidean metric in the output layer of the DL network. This naturally induces two modified versions of the gradient descent flow in the parameter space, one adapted for the overparametrized setting, and the other for the underparametrized setting. In the overparametrized case, we prove that, provided that a rank condition holds, all orbits of the modified gradient descent drive the ${\mathcal L}^2$ cost to its global minimum at a uniform exponential convergence rate; one thereby obtains an a priori stopping time for any prescribed proximity to the global minimum. We point out relations of the latter to sub-Riemannian geometry. Moreover, we generalize the above framework to the situation in which the rank condition does not hold; in particular, we show that local equilibria can only exist if a rank loss occurs, and that generically, they are not isolated points, but elements of a critical submanifold of parameter space.