Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations

Abstract

Deep neural networks (DNNs) have achieved remarkable empirical success, yet the absence of a principled theoretical foundation continues to hinder their systematic development. In this survey, we present differential equations as a theoretical foundation for understanding, analyzing, and improving DNNs. We organize the discussion around three guiding questions: i) how differential equations offer a principled understanding of DNN architectures, ii) how tools from differential equations can be used to improve DNN performance in a principled way, and iii) what real-world applications benefit from grounding DNNs in differential equations. We adopt a two-fold perspective spanning the model level, which interprets the whole DNN as a differential equation, and the layer level, which models individual DNN components as differential equations. From these two perspectives, we review how this framework connects model design, theoretical analysis, and performance improvement. We further discuss real-world applications, as well as key challenges and opportunities for future research.

Figures (4)

• Figure 1: Overview of the survey. This survey is organized around three key questions concerning DNNs through the lens of differential equations: (i) How can differential equations offer a principled understanding of DNN architectures? (ii) How can tools from differential equations be used to improve DNN performance in a principled way? and (iii) What real-world applications benefit from grounding DNNs in differential equations? We focus on two levels of abstraction: the model level, which interprets the entire DNN as a dynamical system, and the layer level, which models individual layers of DNNs as differential equations.
• Figure 2: Summary of early approaches that design DNN architectures through the lens of differential equations. A central idea in these works is to interpret various skip-connection schemes as specific discretizations of ODEs, or introduce different ODE systems for special properties. The plus and minus symbols beside the arrows denote additive or subtractive operations, respectively, and the numbers indicate coefficients.
• Figure 3: Illustration of three commonly used types of differential equations in NDEs. They share the unified formulation $\dd{\bm{h}} = \bm{F}(\bm{h})\dd{\bm{x}}$, where the specific type depends on the regularity of the driving signal $\bm{x}(t)$. (a) When $\bm{x}(t) = t$, the system corresponds to an ODE; (b) when $\bm{x}(t)$ has bounded variation, it becomes a CDE; and (c) when $\bm{x}(t)$ is a Brownian motion, it defines an SDE.
• Figure 4: Comparison between flow-based and diffusion-based generative models. Flow models generate samples by simulating ODEs, resulting in smooth and deterministic sampling trajectories. In contrast, diffusion models rely on SDEs, producing stochastic and inherently rough trajectories due to the presence of noise during the sampling process.

Theorems & Definitions (4)

• Remark 1
• Remark 2
• Remark 3
• Remark 4