Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Neural Differential Equations

Patrick Kidger

TL;DR

This doctoral thesis surveys neural differential equations (NDEs), unifying neural networks with differential equation formalisms to exploit continuous-depth modeling. It systematically covers neural ODEs, neural CDEs, and neural SDEs, detailing existence, training, architectures, interpolation schemes, and numerical solvers, while highlighting universal approximation results and practical tricks like augmentation and reversible solvers. Core contributions include comprehensive theoretical insights into universal approximation for augmented ODEs, development of efficient SDE/CNF frameworks, and extensive discussion of numerical strategies (including backpropagation methods, Hutchinson trace estimation, and batch handling for irregular data). The work also documents public software efforts (Diffrax, torchdiffeq, torchcde, torchsde) and presents concrete demonstrations across physical modelling, irregular time series, and generative modelling, reinforcing NDEs as a versatile toolkit for combining deep learning with dynamical systems. Overall, the thesis positions NDEs as a practical, theory-rich paradigm for scalable, continuous-time modeling in science and engineering, with broad implications for time-series analysis, simulation-informed learning, and probabilistic generation.

Abstract

The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.

On Neural Differential Equations

TL;DR

This doctoral thesis surveys neural differential equations (NDEs), unifying neural networks with differential equation formalisms to exploit continuous-depth modeling. It systematically covers neural ODEs, neural CDEs, and neural SDEs, detailing existence, training, architectures, interpolation schemes, and numerical solvers, while highlighting universal approximation results and practical tricks like augmentation and reversible solvers. Core contributions include comprehensive theoretical insights into universal approximation for augmented ODEs, development of efficient SDE/CNF frameworks, and extensive discussion of numerical strategies (including backpropagation methods, Hutchinson trace estimation, and batch handling for irregular data). The work also documents public software efforts (Diffrax, torchdiffeq, torchcde, torchsde) and presents concrete demonstrations across physical modelling, irregular time series, and generative modelling, reinforcing NDEs as a versatile toolkit for combining deep learning with dynamical systems. Overall, the thesis positions NDEs as a practical, theory-rich paradigm for scalable, continuous-time modeling in science and engineering, with broad implications for time-series analysis, simulation-informed learning, and probabilistic generation.

Abstract

The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.
Paper Structure (468 sections, 32 theorems, 318 equations, 23 figures, 1 table, 10 algorithms)

This paper contains 468 sections, 32 theorems, 318 equations, 23 figures, 1 table, 10 algorithms.

Table of Contents

  1. Introduction
  2. Motivation
  3. Getting started
  4. Prerequisites
  5. Code
  6. Experiments
  7. What is a neural differential equation anyway?
  8. A familiar example
  9. Continuous-depth neural networks
  10. An important distinction
  11. The case for neural differential equations
  12. Applications
  13. Physical (financial, biological, …) modelling
  14. Time series
  15. Generative modelling
  16. ...and 453 more sections

Key Result

Theorem 2.1

Let $f \colon [0, T] \times \mathbb{R}^d \to \mathbb{R}^d$ be continuous in $t$ and uniformly LipschitzThat is, it is Lipschitz in $y$ and the Lipschitz constant is independent of $t$: there exists $C > 0$ such that for all $t, y_1, y_2$ then $\left\|f_\theta(t, y_1) - f_\theta(t, y_2)\right\| \leq

Figures (23)

  • Figure 1: Computation graph for a simple neural ODE.
  • Figure 2: A continuous normalising flow continuously deforms one distribution into another distribution. The flow lines show how particles from the base distribution are perturbed until they approximate the target distribution.
  • Figure 3: Continuous normalising flow example. From top to bottom: target, cat, butterfly. From left to right: the first six pictures show the evolution of the distribution of the CNF as it integrates from $t=0$ to $t=T$, transforming a normal distribution into the desired distribution. The second to last picture shows samples from the learnt CNF. The final picture shows the image used to specify the desired distribution.
  • Figure 4: Overview of latent ODE model.
  • Figure 5: Plots of samples $y \colon [0, 12] \to \mathbb{R}^2$ drawn from the latent ODE model. The leftmost picture is a sample from the untrained model. The rightmost picture is a sample from a fully-trained model. In between are samples from partially-trained models. Quality increases as training proceeds.
  • ...and 18 more figures

Theorems & Definitions (141)

  • Theorem 2.1: Picard's Existence Theorem
  • Remark 2.2
  • Theorem 2.3: Instantaneous change of variables
  • Remark 2.4
  • Remark 2.5
  • Example 2.6
  • Remark 2.7
  • Remark 2.8
  • Remark 2.9
  • Example 2.10
  • ...and 131 more