Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating synthetic data for neural operators

Erisa Hasani, Rachel A. Ward

TL;DR

The paper tackles the data bottleneck in neural PDE solvers by introducing backward data generation, where unknown solutions $u$ from a known Sobolev space are sampled and differentiated to yield the forcing term $f$, enabling exact synthetic training pairs for neural operators. This data is used to train a Fourier Neural Operator to learn the solution operator for a class of elliptic PDEs, with experiments spanning Poisson and semilinear and linear second order PDEs in both Dirichlet and Neumann settings. Results show that models trained solely on synthetic backward data generalize well to data produced by classical solvers, outperforming forward data generation in several non trig RHS scenarios and offering significant speedups in data generation. The approach is architecture agnostic and can potentially be extended to other PDE classes and transfer learning setups, reducing dependence on numerical solvers for training data while maintaining accuracy across varied domains and operators.

Abstract

Recent advances in the literature show promising potential of deep learning methods, particularly neural operators, in obtaining numerical solutions to partial differential equations (PDEs) beyond the reach of current numerical solvers. However, existing data-driven approaches often rely on training data produced by numerical PDE solvers (e.g., finite difference or finite element methods). We introduce a "backward" data generation method that avoids solving the PDE numerically: by randomly sampling candidate solutions $u_j$ from the appropriate solution space (e.g., $H_0^1(Ω)$), we compute the corresponding right-hand side $f_j$ directly from the equation by differentiation. This produces training pairs ${(f_j, u_j)}$ by computing derivatives rather than solving a PDE numerically for each data point, enabling fast, large-scale data generation consisting of exact solutions. Experiments indicate that models trained on this synthetic data generalize well when tested on data produced by standard solvers. While the idea is simple, we hope this method will expand the potential of neural PDE solvers that do not rely on classical numerical solvers to generate their data.

Generating synthetic data for neural operators

TL;DR

The paper tackles the data bottleneck in neural PDE solvers by introducing backward data generation, where unknown solutions from a known Sobolev space are sampled and differentiated to yield the forcing term , enabling exact synthetic training pairs for neural operators. This data is used to train a Fourier Neural Operator to learn the solution operator for a class of elliptic PDEs, with experiments spanning Poisson and semilinear and linear second order PDEs in both Dirichlet and Neumann settings. Results show that models trained solely on synthetic backward data generalize well to data produced by classical solvers, outperforming forward data generation in several non trig RHS scenarios and offering significant speedups in data generation. The approach is architecture agnostic and can potentially be extended to other PDE classes and transfer learning setups, reducing dependence on numerical solvers for training data while maintaining accuracy across varied domains and operators.

Abstract

Recent advances in the literature show promising potential of deep learning methods, particularly neural operators, in obtaining numerical solutions to partial differential equations (PDEs) beyond the reach of current numerical solvers. However, existing data-driven approaches often rely on training data produced by numerical PDE solvers (e.g., finite difference or finite element methods). We introduce a "backward" data generation method that avoids solving the PDE numerically: by randomly sampling candidate solutions from the appropriate solution space (e.g., ), we compute the corresponding right-hand side directly from the equation by differentiation. This produces training pairs by computing derivatives rather than solving a PDE numerically for each data point, enabling fast, large-scale data generation consisting of exact solutions. Experiments indicate that models trained on this synthetic data generalize well when tested on data produced by standard solvers. While the idea is simple, we hope this method will expand the potential of neural PDE solvers that do not rely on classical numerical solvers to generate their data.
Paper Structure (15 sections, 1 theorem, 28 equations, 11 figures, 4 tables)

This paper contains 15 sections, 1 theorem, 28 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Set-up and Main Approach
  3. Drawing synthetic representative functions from a Sobolev space
  4. Representative functions in rectangular domains
  5. Representative functions in non-rectangular domains
  6. Numerical Experiments using the Fourier Neural Operator
  7. The Poisson Equation
  8. Second-order semi-linear elliptic PDE
  9. Comparison to a more classical approach
  10. Second-order linear elliptic PDE
  11. Further examples
  12. Limitations, conclusion and future work
  13. Appendix
  14. Notation
  15. The Darcy flow equation.

Key Result

Theorem 1

The Laplace-Dirichlet operator has countably many eigenvalues $0< \lambda_1 \le \lambda_2 \le \cdots \lambda_N \le \cdots$. There exists an orthonormal basis $(e_i)_{i=0}^{\infty}$ of $L^2(\Omega)$ such that $e_i$ is an eigenfunction of the Laplace-Dirichlet operator, i.e of problem eigenvalue_probl

Figures (11)

  • Figure 1: Predicted solutions of the Poisson equation \ref{['Poisson']} with $f_1(x,y)=x-y$ as the right-hand side using FNO with $1,000$, $10,000$ and $100,000$ training data points. Their relative $L^2$ errors are $0.097$, $0.096$ and $0.094$, respectively, while their relative $l^\infty$ errors are $0.145, 0.123$ and $0.118$, respectively.
  • Figure 2: Predicted solutions of the Poisson equation \ref{['Poisson']} with $f_2(x,y) = |x-0.5||y-0.5|$ as the right-hand side using FNO with $1,000$, $10,000$ and $100,000$ training data points. Their relative $L^2$ errors are $0.102$, $0.114$ and $0.108$, respectively, while their relative $l^\infty$ errors are $0.326, 0.188$ and $0.131$, respectively.
  • Figure 3: Predicted solutions of the semi-linear equation \ref{['semilinear']} with $f_1(x,y)=xy$ as the right-hand side using FNO with $1,000$, $10,000$ and $100,000$ training data points. Their relative $L^2$ errors are $0.058$, $0.058$ and $0.059$, respectively, while their relative $l^\infty$ errors are $0.089, 0.084$ and $0.083$, respectively.
  • Figure 4: Predicted solutions of the semi-linear equation \ref{['semilinear']} with $f_2(x,y) = (x-0.5)^2 + (y-0.5)^2$ as the right-hand side using FNO with $1,000$, $10,000$ and $100,000$ training data points. Their relative $L^2$ errors are $0.107$, $0.115$ and $0.118$, respectively, while their relative $l^\infty$ errors are $0.326, 0.113$ and $0.106$, respectively.
  • Figure 5: Comparison of our backward generation method with the classical forward generation method, as described in Subsection \ref{['subsection:comparison_with_classical_solvers']}, on the semi-linear problem \ref{['semilinear']}. Relative $L^2$ errors of predicted solutions are shown: blue corresponds to our backward method and orange corresponds to the forward method. Our method consistently achieves lower errors across all test cases.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Remark 1
  • Remark 2