Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Neural Operators: Stable Long-Term PDE Prediction

Zaijun Ye, Chen-Song Zhang, Wansheng Wang

TL;DR

This work tackles long-term forecasting for time-dependent PDEs by addressing the train-test mismatch inherent in teacher forcing. It introduces Recurrent Neural Operators (RNOs), which train the operator through autoregressive rollouts, aligning training with inference dynamics. The authors prove that recurrent training reduces worst-case error growth from exponential to linear in the forecast horizon and demonstrate empirically that RNOs, especially r-MgNO, achieve superior long-horizon accuracy and stability compared to teacher-forced baselines and post-hoc refiners. The approach offers a principled path toward robust data-driven dynamics for complex PDE systems, albeit with higher training costs and potential vanishing-gradient challenges that warrant future optimization.

Abstract

Neural operators have emerged as powerful tools for learning solution operators of partial differential equations. However, in time-dependent problems, standard training strategies such as teacher forcing introduce a mismatch between training and inference, leading to compounding errors in long-term autoregressive predictions. To address this issue, we propose Recurrent Neural Operators (RNOs)-a novel framework that integrates recurrent training into neural operator architectures. Instead of conditioning each training step on ground-truth inputs, RNOs recursively apply the operator to their own predictions over a temporal window, effectively simulating inference-time dynamics during training. This alignment mitigates exposure bias and enhances robustness to error accumulation. Theoretically, we show that recurrent training can reduce the worst-case exponential error growth typical of teacher forcing to linear growth. Empirically, we demonstrate that recurrently trained Multigrid Neural Operators significantly outperform their teacher-forced counterparts in long-term accuracy and stability on standard benchmarks. Our results underscore the importance of aligning training with inference dynamics for robust temporal generalization in neural operator learning.

Recurrent Neural Operators: Stable Long-Term PDE Prediction

TL;DR

This work tackles long-term forecasting for time-dependent PDEs by addressing the train-test mismatch inherent in teacher forcing. It introduces Recurrent Neural Operators (RNOs), which train the operator through autoregressive rollouts, aligning training with inference dynamics. The authors prove that recurrent training reduces worst-case error growth from exponential to linear in the forecast horizon and demonstrate empirically that RNOs, especially r-MgNO, achieve superior long-horizon accuracy and stability compared to teacher-forced baselines and post-hoc refiners. The approach offers a principled path toward robust data-driven dynamics for complex PDE systems, albeit with higher training costs and potential vanishing-gradient challenges that warrant future optimization.

Abstract

Neural operators have emerged as powerful tools for learning solution operators of partial differential equations. However, in time-dependent problems, standard training strategies such as teacher forcing introduce a mismatch between training and inference, leading to compounding errors in long-term autoregressive predictions. To address this issue, we propose Recurrent Neural Operators (RNOs)-a novel framework that integrates recurrent training into neural operator architectures. Instead of conditioning each training step on ground-truth inputs, RNOs recursively apply the operator to their own predictions over a temporal window, effectively simulating inference-time dynamics during training. This alignment mitigates exposure bias and enhances robustness to error accumulation. Theoretically, we show that recurrent training can reduce the worst-case exponential error growth typical of teacher forcing to linear growth. Empirically, we demonstrate that recurrently trained Multigrid Neural Operators significantly outperform their teacher-forced counterparts in long-term accuracy and stability on standard benchmarks. Our results underscore the importance of aligning training with inference dynamics for robust temporal generalization in neural operator learning.

Paper Structure

This paper contains 36 sections, 3 theorems, 34 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Operator Learning for Evolution Equations
  4. Challenges in Long-Term Prediction and Extrapolation
  5. Our Perspective: Recurrent Training for Improved Extrapolation
  6. Methodology
  7. Deep Neural Operator Architecture
  8. Recurrent Operator Learning Framework
  9. Theoretical Comparison of Error Propagation
  10. Experiments and Results
  11. Overview of Benchmarks and Baselines
  12. Basic Rollout Performance
  13. Long-Horizon Rollout Performance
  14. Ablation Study
  15. Training Cost
  16. ...and 21 more sections

Key Result

Theorem 1

Let $u_n$ be the exact solution sequence following of the problem eq:base_problem, and let $\hat{u}_n^{TF}$ and $\hat{u}_n^{RNO}$ be the approximate solutions generated using an operator $\mathcal{G}_\theta$ trained via teacher forcing and recurrent training, respectively. Assume the learned operato where $C > 0$ is the Lipschitz constant of the neural operator $\mathcal{G}_\theta$ with respect to

Figures (7)

  • Figure 1: Recurrent Neural Operators. (Left) RNO aligns training with autoregressive inference, unlike teacher forcing which causes a train-test mismatch. (Right) Qualitative comparison on the transient heat conduction problem: teacher-forcing (top), ground truth (middle), and RNO (bottom) predictions with corresponding absolute errors. RNO shows improved long-term stability.
  • Figure 2: Comparison of long-term rollout stability on the Allen–Cahn equation. Both panels show the mean relative $L^2$ error versus rollout time steps. Models were trained for $n=10$ time steps. (a) Performance up to $n=50$ time steps. (b) Performance over an extremely long horizon up to $n=500$ time steps.
  • Figure 3: Ablation study on the Allen-Cahn equation using the r-MgNO model. (a) Effect of varying the number of observation steps during training. (b) Effect of varying the time step size ($\Delta t$).
  • Figure 4: Visualization of predicted solutions for the transient heat conduction problem. The base model is MgNO.
  • Figure 5: Visualization of predicted solutions for the Allen-Cahn problem. The base model is MgNO.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Theorem 1: Error Bounds for Teacher Forcing vs. Recurrent Training
  • Proposition 1
  • Lemma 1