Respecting causality is all you need for training physics-informed neural networks

TL;DR

<3-5 sentence high-level summary>This work identifies the neglect of spatio-temporal causality as a key reason PINNs struggle on chaotic and multi-scale dynamics, and introduces a simple, principled causal training scheme that re-weights the PDE residuals over time. The core idea is to enforce temporal causality with a weighted residual loss $\mathcal{L}_r(\theta) = \frac{1}{N_t} \sum_{i=1}^{N_t} w_i \mathcal{L}_r(t_i, \theta)$ where $w_i = \exp\left(-\epsilon \sum_{k=1}^{i-1} \mathcal{L}_r(t_k, \theta)\right)$, and to adapt $\epsilon$ via annealing while using a stopping criterion based on the temporal weights. The authors demonstrate state-of-the-art accuracy on challenging benchmarks (Lorenz system, Kuramoto–Sivashinsky in chaotic regime, and Navier–Stokes turbulence), achieving 10–100x improvements over prior PINNs in forward simulations. They also provide practical considerations, including exact periodic boundary enforcement via Fourier features, Taylor-mode automatic differentiation for high-order derivatives, and scalable data-parallel training, which broaden the applicability of PINNs to industrially complex problems.

Abstract

While the popularity of physics-informed neural networks (PINNs) is steadily rising, to this date PINNs have not been successful in simulating dynamical systems whose solution exhibits multi-scale, chaotic or turbulent behavior. In this work we attribute this shortcoming to the inability of existing PINNs formulations to respect the spatio-temporal causal structure that is inherent to the evolution of physical systems. We argue that this is a fundamental limitation and a key source of error that can ultimately steer PINN models to converge towards erroneous solutions. We address this pathology by proposing a simple re-formulation of PINNs loss functions that can explicitly account for physical causality during model training. We demonstrate that this simple modification alone is enough to introduce significant accuracy improvements, as well as a practical quantitative mechanism for assessing the convergence of a PINNs model. We provide state-of-the-art numerical results across a series of benchmarks for which existing PINNs formulations fail, including the chaotic Lorenz system, the Kuramoto-Sivashinsky equation in the chaotic regime, and the Navier-Stokes equations in the turbulent regime. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such systems, introducing new opportunities for their applicability to problems of industrial complexity.

Figures (27)

• Figure 1: Allen-Cahn equation:Top: Reference solution versus the prediction of a trained conventional physics-informed neural network. The resulting relative $L^2$ error is $49.87\%$. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at $t=0.0, 0.5, 1.0$.
• Figure 2: Allen-Cahn equation:Left: Loss convergence of training a conventional physics-informed neural network for $2 \times 10^5$ iterations. Middle: Temporal residual loss $\mathcal{L}(t, \bm{\theta})$ at different iteration of the training. Right: Temporal convergent rate at different iteration of the training.
• Figure 3: Allen-Cahn equation:Top: Reference solution versus the prediction of a trained physics-informed neural network using Algorithm \ref{['alg']}. The resulting relative $L^2$ error is $1.43e-03$. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at $t=0.0, 0.5, 1.0$.
• Figure 4: Allen-Cahn equation:Left: Loss convergence of training a physics-informed neural network using Algorithm \ref{['alg']}. Middle: Temporal residual loss $\mathcal{L}(t, \bm{\theta})$ at different iteration of the training. Right: Temporal weights at different iteration of the training.
• Figure 5: Lorentz system: Comparison between the predicted and reference solutions.

Theorems & Definitions (1)

• Definition 2.1