Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations

TL;DR

The paper addresses the gap between theory and practice by proving that the gradient-descent map $G_\eta(\theta)=\theta-\eta\nabla L(\theta)$ is non-singular for neural networks with piecewise analytic activations across realistic architectures (fully connected, convolutional, and attention layers) for almost all step-sizes $\eta$. It does so by showing the empirical loss is almost-everywhere analytic and by establishing a neural-network analogue of the chain rule, ensuring the GD map is invertible except on a measure-zero set of $\eta$; this result extends to SGD via per-step non-singularity and composes to non-singularity of multi-step trajectories. Consequently, prior results on saddle-point avoidance and stability of minima, which rely on non-singularity, now apply in practical deep-learning settings. The findings reinforce the theoretical underpinnings of GD/SGD performance and offer a rigorous basis for analyzing learning dynamics in modern architectures.

Abstract

The theory of training deep networks has become a central question of modern machine learning and has inspired many practical advancements. In particular, the gradient descent (GD) optimization algorithm has been extensively studied in recent years. A key assumption about GD has appeared in several recent works: the \emph{GD map is non-singular} -- it preserves sets of measure zero under preimages. Crucially, this assumption has been used to prove that GD avoids saddle points and maxima, and to establish the existence of a computable quantity that determines the convergence to global minima (both for GD and stochastic GD). However, the current literature either assumes the non-singularity of the GD map or imposes restrictive assumptions, such as Lipschitz smoothness of the loss (for example, Lipschitzness does not hold for deep ReLU networks with the cross-entropy loss) and restricts the analysis to GD with small step-sizes. In this paper, we investigate the neural network map as a function on the space of weights and biases. We also prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures (with fully connected, convolutional, or softmax attention layers) and piecewise analytic activations (which includes sigmoid, ReLU, leaky ReLU, etc.) for almost all step-sizes. Our work significantly extends the existing results on the convergence of GD and SGD by guaranteeing that they apply to practical neural network settings and has the potential to unlock further exploration of learning dynamics.

Figures (4)

• Figure 1: Illustrating the difference between singular and non-singular GD maps (different step-sizes). The loss used is $L(\theta) = 0.5\theta^4-3\theta^2+8 \text{ if } -2 \leq \theta \leq 2; \theta^2 \text{ otherwise}$. Left: for $\eta=0.5$, the GD map is singular. The red interval gets mapped to a single point after one iteration. Right: for $\eta = 0.25$, the GD map is non-singular. The same interval is mapped not to a point, but to an interval.
• Figure 2: Illustration of the main idea in the proof of Proposition \ref{['prop:ff-fcaea']}.
• Figure 3: Left: periodic trajectories for GD on $L(\theta_1, \theta_2) = 3.53(1 - \text{ReLU}(\theta_2 \text{ReLU}(\theta_1)))^2$. As the step-size $\eta$ increases, higher order orbits appear and the lower order ones become unstable. Right: for the same initialisation, but different $\eta$, two trajectories, one that converges and one that oscillates.
• Figure 4: Illustrating the different stable minima for GD and SGD. In blue are the global minima for $L(\theta_1, \theta_2) = 3.53(1 - \text{ReLU}(\theta_2 \text{ReLU}(\theta_1)))^2$; in green are the minima stable for GD; in red those stable for SGD. Left: the stable minima for SGD are a proper subset of the stable minima for GD. Right: the stable minima for SGD and GD do not intersect. This change is due to different step-sizes ( $\eta_{\text{left}} = 0.15$ vs $\eta_{\text{right}} =0.3$) and different probabilities for generating the data (cf. Appendix \ref{['section:experiment']}).

Theorems & Definitions (30)

• Theorem 1: Stochastic Gradient Descent Map for Neural Networks is Non-Singular
• Definition 2: Piecewise and almost eveywhere analytic functions
• Example 3: Some almost everywhere analytic functions
• Remark 4: Non-differentiable points
• Remark 5: Why a.e. analytic functions are problematic
• Proposition 6: Analogue of chain rule for neural networks
• proof : Proof sketch
• Proposition 7: Neural networks are a.e. analytic
• proof : Proof sketch.
• Lemma 8: Composition of a.e. analytic maps