Absence of Closed-Form Descriptions for Gradient Flow in Two-Layer Narrow Networks

TL;DR

The paper investigates whether gradient-flow training dynamics in neural networks admit a closed-form solution. It applies Morales-Ramis differential Galois theory to a simple two-layer narrow network with four parameters, analyzing the variational equations along an integral curve. The main result is that the identity component of the differential Galois group, $G^0$, is non-solvable, proving non-integrability in the meromorphic category and ruling out Liouvillian closed-form trajectories. Consequently, exact analytical descriptions of training dynamics are generally unavailable, reinforcing the necessity of numerical methods and motivating future work on broader architectures and stochastic dynamics.

Abstract

In the field of machine learning, comprehending the intricate training dynamics of neural networks poses a significant challenge. This paper explores the training dynamics of neural networks, particularly whether these dynamics can be expressed in a general closed-form solution. We demonstrate that the dynamics of the gradient flow in two-layer narrow networks is not an integrable system. Integrable systems are characterized by trajectories confined to submanifolds defined by level sets of first integrals (invariants), facilitating predictable and reducible dynamics. In contrast, non-integrable systems exhibit complex behaviors that are difficult to predict. To establish the non-integrability, we employ differential Galois theory, which focuses on the solvability of linear differential equations. We demonstrate that under mild conditions, the identity component of the differential Galois group of the variational equations of the gradient flow is non-solvable. This result confirms the system's non-integrability and implies that the training dynamics cannot be represented by Liouvillian functions, precluding a closed-form solution for describing these dynamics. Our findings highlight the necessity of employing numerical methods to tackle optimization problems within neural networks. The results contribute to a deeper understanding of neural network training dynamics and their implications for machine learning optimization strategies.

Figures (3)

• Figure 1: Examples of $2$-dimensional integrable and non-integrable dynamical systems are presented. In the case of the integrable system with the first integral $\Phi$, trajectories are predictable and constrained within the level set of $\Phi$ (left). In contrast, for non-integrable systems, trajectories lack well-defined orderliness (right).
• Figure 2: A diagram of various dynamical systems. In this diagram, we denote an integrable system as an integrable system in the meromorphic category.
• Figure 3: An illustration of variational equations of dynamical system $\frac{d \bold{x}(t)}{dt} = F( \bold{x} )$. For the integral curve $\gamma(t)$ with the initial point $\bold{p}$, $\tilde{\gamma}(t)$ denotes the perturbed curve with the perturbed initial point $\tilde{\bold{p}}$. Let $\eta(t)$ follow variational equation \ref{['eq:ves']} with the initial condition $\eta(0)=(u_1,...,u_n)$. Then $\varepsilon \eta(t)$ is the first-order approximation (in terms of $\varepsilon$) of the perturbations ($\tilde{\gamma}(t)-\gamma(t)$) of the integral curve $\gamma(t)$.

Theorems & Definitions (55)

• Definition : informal, Liouvillian function
• Definition 1: integral curve
• Definition 2: first integral
• Definition 3: functionally independent
• Definition 4: completely integrable
• Definition 5: B-integrable
• Definition 6: meromorphic function
• Definition 7: integrable in the meromorphic category
• Example 1
• Example 2