Fast-Slow Neural Networks for Learning Singularly Perturbed Dynamical Systems

TL;DR

The paper tackles learning and simulating singularly perturbed dynamical systems with dissipative fast dynamics by introducing the Fast-Slow Neural Network (FSNN), a structure-preserving neural ODE that enforces an attracting slow manifold via a Fenichel-normal-form–inspired coordinate change. FSNN combines an invertible coupling flow for the slow-fast transform, a negative Schur form network for stable fast linear dynamics, and a low-rank bilinear map with IMEX integration to faithfully capture both fast and slow time scales, while enabling a data-driven closure on the slow manifold. The authors prove FSNN’s universal approximation capability for normally stable fast-slow systems and demonstrate accurate learning of slow manifolds, eigenstructure, and long-time slow dynamics on Grad moment, two-scale Lorenz96, and Abraham-Lorentz models. The approach yields stable, efficient, data-driven reduced-order representations suitable for climate and plasma contexts, with potential for full-order model discovery when fast-slow separation is present. Overall, FSNN provides a principled framework to embed invariant-manifold constraints in data-driven dynamical modeling, facilitating reliable extrapolation and reduced-order closures.

Abstract

Singularly perturbed dynamical systems play a crucial role in climate dynamics and plasma physics. A powerful and well-known tool to address these systems is the Fenichel normal form, which significantly simplifies fast dynamics near slow manifolds through a transformation. However, this normal form is difficult to realize in conventional numerical algorithms. In this work, we explore an alternative way of realizing it through structure-preserving machine learning. Specifically, a fast-slow neural network (FSNN) is proposed for learning data-driven models of singularly perturbed dynamical systems with dissipative fast timescale dynamics. Our method enforces the existence of a trainable, attracting invariant slow manifold as a hard constraint. Closed-form representation of the slow manifold enables efficient integration on the slow time scale and significantly improves prediction accuracy beyond the training data. We demonstrate the FSNN on examples including the Grad moment system, two-scale Lorenz96 equations, and Abraham-Lorentz dynamics modeling radiation reaction of electrons.

Figures (11)

• Figure 1: Schematic of the fast-slow neural network (FSNN). The original coordinates are projected into fast-slow coordinates using the invertible coupling flow network. In these coordinates, numerical integration of the Fenichel normal form \ref{['eq:fastslowthm']} is used to advance the solution to later time steps. The inverse function of the invertible coupling flow network is used to convert the trajectories in the fast-slow coordinates into the original coordinate system.
• Figure 1: Schematic of training the fast-slow neural network (FSNN). The network is trained on dynamics data, $\mathcal{D}$, which contains examples of trajectories on both the fast and slow timescales and manifold data, $\mathcal{M}$, which contains examples of solution vectors that lie on the slow manifold. This data interacts both with the full network and the component involving the invertible neural network (INN). The overall loss is a combination of the dynamics loss, which includes $L_2$ losses in the original coordinates ($\mathcal{L}_{\rm system}$), fast coordinates ($\mathcal{L}_{\rm fast}$), and slow coordinates ($\mathcal{L}_{\rm slow}$), and the manifold loss ($\mathcal{L}_{\rm manifold}$), which is an $L_2$ loss of the fast coordinate, which should map to $y=0$ on the slow manifold.
• Figure 1: Demonstration of attracting slow manifold property of the FSNN. For three different random initializations of the FSNN, initial conditions are drawn randomly from a standard normal distribution and are evolved in time to a later time ($t=4$). The dynamics are shown to limit to the curve representing the slow manifold.
• Figure 2: Plot showing the eigenvalues encountered in various regions of the $m$-$p$ plane, where $m = \frac{1}{2} {\rm Tr} (\mathcal{M}) = R \cos(\theta)$ and $p={\rm det}(\mathcal{M}) = R^2 - r^2$. The green region corresponds to linear stability, ${\rm Re}(\lambda_+, \lambda_-) < 0$, and the red region corresponds to linear instability, ${\rm Re}(\lambda_+, \lambda_-) > 0$. Above the parabola $p=m^2$ (in the blue region), eigenvalues are complex. Below the parabola (outside of the blue region), the eigenvalues are real.
• Figure 2: Results of learning the dynamics of the simple attracting manifold example described by \ref{['eq:toy']}. Left three plots on top row: phase portraits comparing the ground truth (g.t.) reference solution (bold curves) and that of the trained model (dots) for $\epsilon=0, 0.001, 0.01$. Left three plots on bottom row: corresponding errors in time for each trajectory in the above phase portraits. Rightmost plot on top row: comparison of the eigenvalue of the negative Schur form as a function of $z_1$ learned by the FSNN with the ground truth eigenvalue given by $\lambda$ in \ref{['eq:evalsman']}. Rightmost plot on bottom row: error between the learned slow manifold and the ground truth slow manifold given by $\theta$ in \ref{['eq:evalsman']}.

Theorems & Definitions (22)

• Definition 1
• Definition 2
• Definition 3
• Lemma 4
• Lemma 5
• Theorem 6
• Definition 7
• Theorem 8
• Theorem 9
• Definition 1