SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

TL;DR

This work analyzes gradient-flow dynamics for fully connected networks with smooth, definable activations using o-minimal structures. It proves a dichotomy: gradient flows either converge to a critical point or diverge to infinity while the loss converges to a generalized critical value, and establishes a threshold $oldsymbol{}$ ensuring convergence to the global infimum from near-optimal starts. For polynomial targets of degree at least two, with large enough networks and data, the loss has infimum zero but no global minimum, implying asymptotic realization and gradient-flow divergence from good initializations; the SAD activation class and definability are central to these results. Numerical experiments with polynomial targets, PDE solvers (Deep Kolmogorov), and MNIST corroborate the theoretical predictions and suggest the divergence phenomenon holds beyond idealized settings. The paper advances theoretical understanding of training dynamics under smooth, definable activations and highlights implications for optimization in regimes where loss can approach zero without attaining a true minimum.

Abstract

We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold $\varepsilon>0$ such that the loss value of any gradient flow initialized at most $\varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.

Figures (6)

• Figure 1: Approximation of polynomial target functions using different activation functions and GD algorithm. From left to right: 1-dimensional input, 2-dimensional input, 4-dimensional input.
• Figure 2: Approximation of polynomial target functions using different activation functions and Adam algorithm. From left to right: 1-dimensional, 2-dimensional, and 4-dimensional input.
• Figure 3: Left: approximation of Heat PDE and Black-Scholes PDE solutions using the deep Kolmogorov method. Right: image classification on the MNIST dataset.
• Figure 4: Structure of the special cases in the proof of \ref{['polynomialapprox']}.
• Figure 5: An illustration how a neural network with 3 hidden layers can realize functions of the form $f_2\circ g \circ f_1$ where $g$ is the realization of a shallow neural network with $n$ neurons and $f_i$ are realizations of shallow neural networks with on hidden neuron. Dotted lines correspond to weights that are set to zero.

Theorems & Definitions (46)

• Definition 1: O-minimal structure
• Definition 2: Pfaffian functions
• Theorem 2.1: Wilkie1999
• Example 1
• Definition 3: Neural network architectures
• Definition 4: Generalized critical values
• Definition 5: Locally Lipschitz derivative
• Theorem 2.2: Dichotomy for gradient flows
• proof
• Remark 1