Neural Emulator Superiority: When Machine Learning for PDEs Surpasses its Training Data

TL;DR

This work defines and analyzes emulator superiority, the phenomenon where neural emulators trained on low-fidelity solver data can outperform the training solver when measured against a high-fidelity reference. It develops a formal framework with a rollout-based superiority metric $\xi^{[t]}$ and distinguishes state-space from autoregressive superiority, supported by closed-form proofs for linear PDEs via Fourier analysis and extensive nonlinear experiments. The authors demonstrate that emulators can learn more regularized or favorable error propagation properties, enabling higher physical fidelity in specific regimes, including advection, diffusion, Poisson, and Burgers equations. These findings invite a re-evaluation of emulator benchmarking and suggest that carefully designed inductive biases and rollout dynamics can yield practical speedups and accuracy gains beyond the training data limitations.

Abstract

Neural operators or emulators for PDEs trained on data from numerical solvers are conventionally assumed to be limited by their training data's fidelity. We challenge this assumption by identifying "emulator superiority," where neural networks trained purely on low-fidelity solver data can achieve higher accuracy than those solvers when evaluated against a higher-fidelity reference. Our theoretical analysis reveals how the interplay between emulator inductive biases, training objectives, and numerical error characteristics enables superior performance during multi-step rollouts. We empirically validate this finding across different PDEs using standard neural architectures, demonstrating that emulators can implicitly learn dynamics that are more regularized or exhibit more favorable error accumulation properties than their training data, potentially surpassing training data limitations and mitigating numerical artifacts. This work prompts a re-evaluation of emulator benchmarking, suggesting neural emulators might achieve greater physical fidelity than their training source within specific operational regimes. Project Page: https://tum-pbs.github.io/emulator-superiority

Figures (13)

• Figure 1: A PDE can be solved using different numerical algorithms. A coarse solver $\mathcal{P}$ produces data trajectories with inherent numerical errors (a). Neural emulators have inductive biases that enable them to learn a regularized, more physical model $f_\theta$ (b). When we measure the accuracy of both emulator $f_\theta$ and coarse simulator $\mathcal{P}$ against a higher fidelity ground truth $\hat{\mathcal{P}}$ (c), the networks can exceed the training reference, giving rise to "emulator superiority" (d).
• Figure 2: Both explicit $\hat{\mathbf{e}}_h$ and implicit $\hat{\iota}_h$ schemes introduce an error in the modes' magnitude either in dampening (reducing) or unstably amplifying (increasing) it. The former is visible as numerical diffusion in state space (a). This error is more pronounced for higher modes $\phi$ and larger $|\gamma_1|$. For low $\phi < 0.25$, the explicit scheme has less numerical diffusion. For higher $\phi$, the implicit scheme is better. The explicit scheme incurs a stability restriction $|\gamma_1|<1$ to be stable for all modes. Similarly, the explicit scheme is better in terms of phase error within stability limits (b).
• Figure 3: The superiority ratio for a simple two-parameter ansatz when trained on data generated by an implicit first-order upwind fitted at relative mode $\psi = M/N \in [0, 1/2]$ and tested across relative mode $\phi = m/N \in [0, 1/2]$. $M$ is the one mode at which we fit the emulator, $m$ is the one mode we test it at, and $N$ is the spatial resolution. A value below $1$ indicates that the learned method is superior to its training simulator. Superiority occurs in both magnitude and phase error "forwardly" in that we are superior for a region with $\phi > \psi$.
• Figure 4: Superiority of an emulator fitted on relative mode $\psi = M/N \in [0, 1/2]$ for the diffusion (across different difficulties/intensities $\gamma_2$) and Poisson equation (across different numbers of iterations $q$ of the Richardson iteration). Similar to the advection equation (figure \ref{['fig:implicit_implicit_superiority_advection_magnitude_and_phase']}), diffusion displays superiority forwardly, i.e., $\phi > \psi$ whereas Poisson has backward superiority, i.e., $\phi < \psi$.
• Figure 5: Almost all considered emulator architectures achieve autoregressive superiority when the IC distribution does not change from training to test (a) & (b). Hence, superiority can also be achieved against the better explicit scheme within its stability limit (b). State-space superiority (under varied IC distribution) is only possible with the favorable local inductive bias of a ConvNet (c).