High-order expansion of Neural Ordinary Differential Equations flows

TL;DR

Event Transition Tensors is introduced, a framework based on high-order differentials that provides a rigorous mathematical description of neural ODE dynamics on event manifolds that contributes to a deeper theoretical foundation for event-triggered neural differential equations and provides a mathematical construct for explaining complex system dynamics.

Abstract

Artificial neural networks, widely recognised for their role in machine learning, are now transforming the study of ordinary differential equations (ODEs), bridging data-driven modelling with classical dynamical systems and enabling the development of infinitely deep neural models. However, the practical applicability of these models remains constrained by the opacity of their learned dynamics, which operate as black-box systems with limited explainability, thereby hindering trust in their deployment. Existing approaches for the analysis of these dynamical systems are predominantly restricted to first-order gradient information due to computational constraints, thereby limiting the depth of achievable insight. Here, we introduce Event Transition Tensors, a framework based on high-order differentials that provides a rigorous mathematical description of neural ODE dynamics on event manifolds. We demonstrate its versatility across diverse applications: characterising uncertainties in a data-driven prey-predator control model, analysing neural optimal feedback dynamics, and mapping landing trajectories in a three-body neural Hamiltonian system. In all cases, our method enhances the interpretability and rigour of neural ODEs by expressing their behaviour through explicit mathematical structures. Our findings contribute to a deeper theoretical foundation for event-triggered neural differential equations and provide a mathematical construct for explaining complex system dynamics.

Figures (5)

• Figure 1: Schematics of the high-order expansion of a NeuralODE flow. The dynamics of a system, controlled as well as free, is learned from data resulting in a highly parametrized phase portrait which is constructed to be differentiable everywhere by the choice of activation functions (left). The resulting highly non-linear and complex map from initial conditions to the conditions at some event manifold are represented, up to high-order, via the Event Transition Tensors (right).
• Figure 2: Learning dynamics $\dot{\pmb{x}}$ from observations in the Lotka Volterra equations. a) Time evolutions of uncontrolled and controlled NeuralODEs, where the dynamics are learned from observations. Each trajectory starts at the initial state (population of Hares and Lynx) and reaches the desired target populations in a time-optimal fashion by hunting (or staying on standby) over specific time windows. On the left is the nominal trajectory around which all Taylor expansions in b) and c) are computed. On the right are examples of other time-optimal trajectories with different initial and final populations of Hares and Lynx. b) Deviations in initial populations can be mapped directly to the corresponding optimal start of the hunting season. For instance, if the initial populations of Hares and Lynx in the nominal case are off by +50 and +80, respectively, the optimal start of the hunting season is 72 days after the nominal starting date. c) An uncertainty distribution in the initial state (here Gaussian) can be mapped directly to the statistical moments of the distribution of optimal start of the hunting season. Any probability distribution for which moment generating functions exists can be used to describe the uncertainty distribution in the initial state.
• Figure 3: Flow Expansion of Neural Hamiltonian ODE for Didymos' Irregular Gravitational Field. a) The upper section illustrates the learning framework, where a Neural Hamiltonian model is trained to approximate the irregular gravitational field of Didymos. The total Hamiltonian is expressed as the sum of the Circular Restricted Three-Body Problem (CR3BP) Hamiltonian ($\mathcal{H}_{CR3BP}$) and a neural network term ($\mathcal{N}_{\pmb{\theta}}$). The neural network is trained to capture deviations from the CR3BP model. The lower section represents the NeuralODE-based training setup, which backpropagates errors to refine the learned Hamiltonian. b) The diagram presents a self-stabilizing terminator orbit (SSTO) in blue and a landing trajectory (gray) that connects the SSTO to Dimorphos’ surface. The Juventas CubeSat is also depicted. c) A 3D visualization of the landing trajectory from b), illustrating the spatial distribution of uncertainties at the event surface, defined at a fixed altitude above Dimorphos. The color map on the landing points indicates deviations from the nominal landing position.
• Figure 4: Mass-optimal, pinpoint spacecraft landing on the comet 67P/Churyumov-Gerasimenko using a Guidance & Control Network (G&CNET). The optimal control policy is learned via behavioral cloning on a dataset of optimal trajectories (not shown in this figure). a) The nominal neurocontrolled trajectory is shown from various perspectives in a rotating frame in which the asteroid is kept fixed. On the top right is the G&CNET architecture, taking in as inputs the state of the spacecraft (position $[x,y,z]$, velocity $[v_x,v_y,v_z]$ and mass $m$) and outputting the corresponding optimal controls (throttle $u$ and thrust direction $[i_x,i_y,i_z]$). The G&CNET's performance can be evaluated by computing how a given distribution in initial state uncertainty b) propagates to a specific event manifold c). The event manifold used for this case is parameterized with a feedforward neural network and represents the boundary at $250$m in altitude above the surface of the comet. The color of the points on the event manifold represents the local density of points, with red corresponding to the highest density and blue to the lowest.
• Figure 5: Energy-optimal drone racing through a square gate using a Guidance & Control Network (G&CNET). The optimal control policy is learned via behavioral cloning on a dataset of optimal trajectories (not shown in this figure). a) The G&CNET architecture, taking in as inputs the full state vector of the drone and outputting the corresponding optimal controls. b) The G&CNET's performance can be evaluated by computing how a given distribution in initial state uncertainty propagates to a specific event manifold. For instance, it can verified that, given an initial state uncertainty of $\pm0.5$m in position and $\pm1$m/s in velocity, the final distance to the center of the gate at the event, $d_{\text{center}}$, is less than $10$cm in $98.1\%$ of cases. The event manifold used in this case is a simple two-dimensional square gate. The color of the points on the event manifold represents the local density of points, red corresponding to the highest density and blue to the lowest.