Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints

TL;DR

The paper tackles long-horizon learning of constrained dynamical systems by enforcing hard algebraic constraints at every integration step. It introduces Manifold-Projected Neural ODEs (PNODEs), which project tentative updates onto the constraint manifold $\mathcal{M}=\{u:g(u,t)=0\}$ using either a robust iterative solver or a fast single-Jacobian variant, all differentiable via implicit differentiation. The approach unifies and extends prior relaxation and stabilization methods and yields dramatically smaller constraint violations (often $<10^{-10}$) while remaining competitive or faster than baselines across six benchmark systems. Empirical results demonstrate superior long-horizon stability and physical consistency, highlighting explicit projection as a simple, scalable principle for learning constrained dynamics. The work has practical impact for SciML tasks requiring hard invariants (e.g., energy, momentum) in physics-based simulations and robotics, enabling reliable, physically plausible predictions over long time horizons.

Abstract

Despite the promise of scientific machine learning (SciML) in combining data-driven techniques with mechanistic modeling, existing approaches for incorporating hard constraints in neural differential equations (NDEs) face significant limitations. Scalability issues and poor numerical properties prevent these neural models from being used for modeling physical systems with complicated conservation laws. We propose Manifold-Projected Neural ODEs (PNODEs), a method that explicitly enforces algebraic constraints by projecting each ODE step onto the constraint manifold. This framework arises naturally from semi-explicit differential-algebraic equations (DAEs), and includes both a robust iterative variant and a fast approximation requiring a single Jacobian factorization. We further demonstrate that prior works on relaxation methods are special cases of our approach. PNODEs consistently outperform baselines across six benchmark problems achieving a mean constraint violation error below $10^{-10}$. Additionally, PNODEs consistently achieve lower runtime compared to other methods for a given level of error tolerance. These results show that constraint projection offers a simple strategy for learning physically consistent long-horizon dynamics.

Figures (9)

• Figure 1: Qualitative comparison of learned dynamics across neural ODE variants. We show predicted trajectories from various neural ODE models on two benchmark problems: (a) Lotka-Volterra predator-prey dynamics and (b) a 2D mass-spring system with conserved energy. Vanilla neural ODEs diverge over time due to lack of constraint enforcement. Adding soft penalties improves short-term behavior but fails to prevent long-term drift. Stabilized NODEs (SNODEs) reduce drift but still show degradation over extended horizons. In contrast, PNODEs stay aligned with the ground truth throughout, accurately preserving both dynamics and algebraic invariants. These results highlight our key insight: explicit projection onto the constraint manifold enables qualitatively superior and physically consistent learning.
• Figure 2: Overview of the Projected Neural ODE Architecture. Our model augments a standard Neural ODE solver by adding a constraint projection step. At each integration step, the neural network predicts an unconstrained update, which is then projected back onto the constraint manifold using a nonlinear solver. This projection enforces algebraic constraints explicitly, ensuring the system evolves on the constraint manifold. The result is a semi-explicit neural DAE framework that combines data-driven dynamics with rigorous constraint satisfaction.
• Figure 3: Training Pipeline for Neural ODEs: We train the model using overlapping time sequences, enabling efficient mini-batching without requiring collocation losses. Each batch serves as a short rollout, while continuity across sequences (from overlapping time sequence) allows the model to learn long-horizon dynamics. In contrast, multiple shooting methods optimize trajectory segments sequentially, requiring explicit continuity constraints. Our training setup enables easy batching and avoids the complexity of enforcing inter-segment consistency during optimization.
• Figure 4: Accuracy & Constraint Satisfaction vs Inference Time: Each point represents a method's performance on a specific system, showing relative state error (top), constraint violation (bottom), and inference time (x-axis). PNODE variants consistently achieve lower runtime compared to other methods for a given level of mean error.
• Figure 5: Projected Neural ODE has the lowest constraint violations on six benchmark systems. We report the relative state error (left axis) and absolute constraint violation (right axis) over long integration horizons for each method: Vanilla NODE, NODE with Soft Constraints, Stabilized NODE (with $\gamma$ = $0.5$, $2.0$, $10.0$), and Projected NODEs (with and w/o multiple Jacobian factorizations). Each subplot corresponds to a different dynamical system -- Lotka-Volterra, mass-spring, two-body, nonlinear spring, robot arm, and rigid body. PNODEs consistently maintain several magnitudes lower constraint violations while achieving competitive or superior prediction accuracy.