Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations

TL;DR

This work addresses solving high-dimensional evolution equations by replacing global time-space training with Neural Galerkin schemes that evolve neural-network parameters in time according to the Dirac–Frenkel variational principle. The key idea is to represent the solution as $U( heta(t),\boldsymbol{x})$ and derive a system $M(\theta)\dot{\theta}=F(t,\theta)$, enabling sequential updates and adaptive time stepping. Crucially, the residuals and their estimations are computed with time-adaptive measures $\nu_{\theta}$ and, when beneficial, using importance sampling to reduce variance. Numerical experiments on Korteweg–de Vries, Allen–Cahn, high-dimensional advection, and Fokker–Planck systems show that active, data-informed sampling combined with nonlinear, time-dependent parametrizations dramatically improves accuracy over traditional, globally trained or uniformly sampled methods. The approach offers a scalable framework for high-dimensional PDEs and kinetic-type equations, with potential impact on simulations of localized traveling waves, particle systems, and related processes where classical solvers struggle.

Abstract

Deep neural networks have been shown to provide accurate function approximations in high dimensions. However, fitting network parameters requires informative training data that are often challenging to collect in science and engineering applications. This work proposes Neural Galerkin schemes based on deep learning that generate training data with active learning for numerically solving high-dimensional partial differential equations. Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations. This is in contrast to other machine learning methods that aim to fit network parameters globally in time without taking into account training data acquisition. Our finding is that the active form of gathering training data of the proposed Neural Galerkin schemes is key for numerically realizing the expressive power of networks in high dimensions. Numerical experiments demonstrate that Neural Galerkin schemes have the potential to enable simulating phenomena and processes with many variables for which traditional and other deep-learning-based solvers fail, especially when features of the solutions evolve locally such as in high-dimensional wave propagation problems and interacting particle systems described by Fokker-Planck and kinetic equations.

Figures (13)

• Figure 1: Korteweg-De Vries: Neural Galerkin schemes integrate in time a nonlinear parametrization of the solution and so obtain an accurate approximation with few degrees of freedom in this example. In contrast, linear (standard) Galerkin that derives approximations in fixed spaces with fixed bases without feature adaptation leads to poor approximations when there are local dynamics in the spatial domain.
• Figure 2: KdV: Basis functions used for linear Galerkin approximations.
• Figure 3: Allen-Cahn: The proposed Neural Galerkin approach with a three-hidden-layer network correctly predicts the dynamics of the sharp walls that eventually separate the solution into two flat pieces at $0$ and $+1$ in this experiment. In contrast, linear Galerkin with the same number of degrees of freedom wrongly predicts a solution with three flat pieces at $-1, 0$, and $+1$ because of the lack of expressiveness of the linear approximation.
• Figure 4: Allen-Cahn: The plots visualize how the used Neural Galerkin scheme propagates forward in time the coefficients and features of a DNN parametrization by following the dynamics prescribed by the PDE. Notice that the output components, defined in Eq. \ref{['eq:ACOutputComponent']} and shown in (a) and (b), reflect the transition of the sharp walls between the flat pieces of the solution and that the dynamics rapidly change from time $t = 1.05$ to about $t = 2.5$ when the potential changes sign in parts of the spatial domain.
• Figure 5: High-dimensional advection with time-only varying advection speed: (a) marginal in dimension 5 of the analytic solution; (b) approximation obtained with static sampling; (c) result of the Neural Galerkin method that adaptively samples data over time to estimate the operators $M$ and $F$ depending on the dynamics of the problem at hand. This is in stark contrast to classical time-space collocation approaches that uniformly sample over space and time when optimizing for a solution. In this example, the adaptive sampling is key for Neural Galerkin to accurately predict the local-in-space dynamics of the solution. In contrast, an approximate solution obtained with uniform sampling that is static over time fails to lead to meaningful predictions, as shown by plot (b) and the error shown in plot (d).

Theorems & Definitions (1)

• Proposition 1