Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows

Shuhao Cao, Francesco Brarda, Ruipeng Li, Yuanzhe Xi

TL;DR

This work tackles accurate and efficient neural operator learning for turbulent Navier-Stokes equations by introducing SpatioTemporal-FNO (ST-FNO), a spatiotemporal Fourier operator that maps between Bochner spaces to enable arbitrary-length temporal super-resolution. A novel hybrid training paradigm is proposed: train the ST-FNO briefly and then fine-tune only the final spectral layer using a functional-type a posteriori error estimator with a negative Sobolev loss, leveraging Parseval to compute the residual exactly. Compared with end-to-end training and conventional solvers, this approach reduces training cost while achieving higher accuracy on NSE benchmarks, by integrating traditional PDE insights (auto-differentiable solvers for extra fields and a convex residual loss) with neural operators. The method demonstrates strong performance on Taylor-Green vortex and 2D isotropic turbulence benchmarks and is supported by open-source code and data, highlighting a practical pathway to physics-informed, mesh-free turbulence simulation.

Abstract

Recent advancements in operator-type neural networks have shown promising results in approximating the solutions of spatiotemporal Partial Differential Equations (PDEs). However, these neural networks often entail considerable training expenses, and may not always achieve the desired accuracy required in many scientific and engineering disciplines. In this paper, we propose a new learning framework to address these issues. A new spatiotemporal adaptation is proposed to generalize any Fourier Neural Operator (FNO) variant to learn maps between Bochner spaces, which can perform an arbitrary-length temporal super-resolution for the first time. To better exploit this capacity, a new paradigm is proposed to refine the commonly adopted end-to-end neural operator training and evaluations with the help from the wisdom from traditional numerical PDE theory and techniques. Specifically, in the learning problems for the turbulent flow modeled by the Navier-Stokes Equations (NSE), the proposed paradigm trains an FNO only for a few epochs. Then, only the newly proposed spatiotemporal spectral convolution layer is fine-tuned without the frequency truncation. The spectral fine-tuning loss function uses a negative Sobolev norm for the first time in operator learning, defined through a reliable functional-type a posteriori error estimator whose evaluation is exact thanks to the Parseval identity. Moreover, unlike the difficult nonconvex optimization problems in the end-to-end training, this fine-tuning loss is convex. Numerical experiments on commonly used NSE benchmarks demonstrate significant improvements in both computational efficiency and accuracy, compared to end-to-end evaluation and traditional numerical PDE solvers under certain conditions. The source code is publicly available at https://github.com/scaomath/torch-cfd.

Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows

TL;DR

This work tackles accurate and efficient neural operator learning for turbulent Navier-Stokes equations by introducing SpatioTemporal-FNO (ST-FNO), a spatiotemporal Fourier operator that maps between Bochner spaces to enable arbitrary-length temporal super-resolution. A novel hybrid training paradigm is proposed: train the ST-FNO briefly and then fine-tune only the final spectral layer using a functional-type a posteriori error estimator with a negative Sobolev loss, leveraging Parseval to compute the residual exactly. Compared with end-to-end training and conventional solvers, this approach reduces training cost while achieving higher accuracy on NSE benchmarks, by integrating traditional PDE insights (auto-differentiable solvers for extra fields and a convex residual loss) with neural operators. The method demonstrates strong performance on Taylor-Green vortex and 2D isotropic turbulence benchmarks and is supported by open-source code and data, highlighting a practical pathway to physics-informed, mesh-free turbulence simulation.

Abstract

Recent advancements in operator-type neural networks have shown promising results in approximating the solutions of spatiotemporal Partial Differential Equations (PDEs). However, these neural networks often entail considerable training expenses, and may not always achieve the desired accuracy required in many scientific and engineering disciplines. In this paper, we propose a new learning framework to address these issues. A new spatiotemporal adaptation is proposed to generalize any Fourier Neural Operator (FNO) variant to learn maps between Bochner spaces, which can perform an arbitrary-length temporal super-resolution for the first time. To better exploit this capacity, a new paradigm is proposed to refine the commonly adopted end-to-end neural operator training and evaluations with the help from the wisdom from traditional numerical PDE theory and techniques. Specifically, in the learning problems for the turbulent flow modeled by the Navier-Stokes Equations (NSE), the proposed paradigm trains an FNO only for a few epochs. Then, only the newly proposed spatiotemporal spectral convolution layer is fine-tuned without the frequency truncation. The spectral fine-tuning loss function uses a negative Sobolev norm for the first time in operator learning, defined through a reliable functional-type a posteriori error estimator whose evaluation is exact thanks to the Parseval identity. Moreover, unlike the difficult nonconvex optimization problems in the end-to-end training, this fine-tuning loss is convex. Numerical experiments on commonly used NSE benchmarks demonstrate significant improvements in both computational efficiency and accuracy, compared to end-to-end evaluation and traditional numerical PDE solvers under certain conditions. The source code is publicly available at https://github.com/scaomath/torch-cfd.
Paper Structure (53 sections, 9 theorems, 83 equations, 13 figures, 7 tables, 2 algorithms)

This paper contains 53 sections, 9 theorems, 83 equations, 13 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Compromises and drawbacks of traditional numerical methods.
  3. Spatiotemporal operator learning.
  4. Limits and the lack of accuracy for neural operators.
  5. New hybrid scheme.
  6. Fine-tuning using functional-type a posteriori error estimation.
  7. Main contributions.
  8. Spatiotemporal Operator Learning for Navier-Stokes Equations
  9. Spatiotemporal Operator learning problem for NSE
  10. Spatiotemporal Adaptation of Fourier Neural Operators
  11. Modifications to 3D FFT.
  12. Modifications to the lifting operator $P$.
  13. Modifications to the projection operator $Q$.
  14. New Hybrid Paradigm for Spatiotemporal Operator Learning
  15. A posteriori error estimation using a functional norm
  16. ...and 38 more sections

Key Result

Theorem 3.1

Let the weak solution to eq:velocity-pressure be $\bm{u} \in L^2(\mathcal{T}; \mathcal{V})$, and $\partial_t \bm{u} \in L^{2} (\mathcal{T}; \mathcal{V}')$. For $\bm{u}$ that is sufficiently regular, the dual norm of the residual is efficient to estimate the error for any $\bm{u}_{\mathcal{N}}$: Moreover, if $\bm{u}$ and $\bm{u}_{\mathcal{N}}$ are "sufficiently close", then it is reliable to serve

Figures (13)

  • Figure 1: Schematic differences between approaches. 4th-order Runge-Kutta (RK4): small time steps bounded by the CFL condition, de-aliasing filter needed; autoregressive NO rolling-out: using previous evaluation as input repetitively, large time steps, no stability guarantees; Spatiotemporal FNO (ST-FNO) with hybrid fine-tuning: large time steps, yielding arbitrary-length temporal prediction in a single evaluation, parallel-in-time optimization for spectral fine-tuning.
  • Figure 2: The FNO3d in li2020fourier is a snapshot learner. : spectral convolution layer ; : pointwise nn.Conv3d that works as channel expansion/reduction; : pointwise nonlinearity. The TikZ source code to produce this figure is modified from the examples in 2018PlotNeuralNet.
  • Figure 3: The Spatiotemporal-adaptated FNO3d (ST-FNO3d) is now a trajectory-to-trajectory learner. : layer normalization to replace a hard-coded global normalization. Combined with channel mixing, the first spectral convolution layer serves as a time-depth-wise separable (global) convolution, after which the time dimension is shrank to a fixed "latent" time dimension through iFFT's resampling. : the spatiotemporal spectral convolution layer as the final layer is fine-tuned after the training phase.
  • Figure 4: Ground truth streamlines for Taylor-Green vortex example.
  • Figure 5: Contours plots of pointwise values of residuals for Example \ref{['list:example-fno3d']}. \ref{['fig:residual-gt']}: the residual of the ground truth; \ref{['fig:residual-fno']}: residual of ST-FNO, where the time derivative in the residual is using the ground truth's; \ref{['fig:residual-ft']}: the residual after fine-tuning for 10 ADAM iterations where the time derivative is computed using an extra-fine-step numerical solver.
  • ...and 8 more figures

Theorems & Definitions (20)

  • Theorem 3.1: A posteriori error bound for any fine-tuned approximations, informal version
  • Theorem 3.2: Functional norm "$\simeq$" negative norm
  • Lemma E.2: Skew-symmetry of the trilinear term
  • proof
  • Lemma E.3: Poincaré inequality
  • proof
  • Lemma E.4: Contuinity and embedding results for the convection term
  • proof
  • Lemma E.5: Energy stability of NSE in a bounded domain
  • proof
  • ...and 10 more