Neural DAEs: Constrained neural networks

TL;DR

This work addresses how to preserve algebraic constraint information in neural network models of dynamical systems by embedding constraint data into architectures rather than relying on data alone. It introduces four approaches—auxiliary regularization, end constraints, smooth constraints, and stabilization by penalty—to enforce constraints expressed as $c(K z(tau)) = 0$, demonstrating their effectiveness on multi-body pendulums, water MD simulations, and divergence-free vector-field denoising. Across these domains, projection-based methods, especially when applied smoothly through the network, yield the strongest improvements in constraint satisfaction and often improve predictive accuracy, highlighting the practical impact of constraint-aware learning in physics-informed contexts. The results suggest a robust, architecture-agnostic strategy for integrating hard physical constraints into neural models, with tradeoffs in computational cost and implementation complexity and clear directions for future refinement, such as advanced solvers, analytic backpropagation for projections, and higher-order constraint information.

Abstract

This article investigates the effect of explicitly adding auxiliary algebraic trajectory information to neural networks for dynamical systems. We draw inspiration from the field of differential-algebraic equations and differential equations on manifolds and implement related methods in residual neural networks, despite some fundamental scenario differences. Constraint or auxiliary information effects are incorporated through stabilization as well as projection methods, and we show when to use which method based on experiments involving simulations of multi-body pendulums and molecular dynamics scenarios. Several of our methods are easy to implement in existing code and have limited impact on training performance while giving significant boosts in terms of inference.

Figures (22)

• Figure 1: A multi-body pendulum system with four pendulums.
• Figure 1: Snapshots of a five-body pendulum. These should be viewed as snapshots of a single pendulum animation. Each snapshot is 100 steps after the previous one (0.1). The arrows indicate the velocity of each individual pendulum.
• Figure 1: Comparison of neural networks using penalty stabilization with different strengths $\gamma$. All neural networks are trained on 100 samples with $k=100$ on the multi-body pendulum system described in Section \ref{['sec:experiment_pendulum']}. Each run has been repeated three times, the solid/dashed lines are the average based on validation/training data. Note that the blue line is mostly hidden beneath the orange line
• Figure 1: Comparison of neural networks using auxiliary regularization with different strengths $\eta$. All neural networks are trained on 100 samples with $k=100$ on the five-body pendulum system described in Section \ref{['sec:experiment_pendulum']}. Each run has been repeated three times, the solid/dashed lines are the average based on validation/training data. Note that the blue line is mostly hidden beneath the orange line.
• Figure 1: Comparison of the different ways of adding constraints to neural network trained on 100 multi-body pendulum samples with $k=100$. Left: Mean absolute error. Right: Maximum constraint violation. Each run has been repeated three times, the solid/dashed lines are the average based on validation/training data.