The Cost-Accuracy Trade-Off In Operator Learning With Neural Networks

TL;DR

This study systematically compares four neural operator learning approaches—PCA-Net, DeepONet, PARA-Net, and Fourier Neural Operator (FNO)—across four PDE problems to quantify how online evaluation cost scales with accuracy under varying data volumes and network sizes. By isolating parameterization and data-volume effects, the work reveals that PARA-Net is consistently less competitive, while FNO often delivers the best cost-accuracy trade-off in 2D problems, albeit with scalability challenges in higher dimensions. The results highlight the importance of output-space representation (fixed PCA bases vs learned trunk functions) and underscore the need for theoretical analyses of cost-accuracy curves in nonlinear operator learning. Overall, the paper provides practical guidance on selecting operator surrogates for PDEs and motivates future theory on the complexity of neural operator methods.

Abstract

The term `surrogate modeling' in computational science and engineering refers to the development of computationally efficient approximations for expensive simulations, such as those arising from numerical solution of partial differential equations (PDEs). Surrogate modeling is an enabling methodology for many-query computations in science and engineering, which include iterative methods in optimization and sampling methods in uncertainty quantification. Over the last few years, several approaches to surrogate modeling for PDEs using neural networks have emerged, motivated by successes in using neural networks to approximate nonlinear maps in other areas. In principle, the relative merits of these different approaches can be evaluated by understanding, for each one, the cost required to achieve a given level of accuracy. However, the absence of a complete theory of approximation error for these approaches makes it difficult to assess this cost-accuracy trade-off. The purpose of the paper is to provide a careful numerical study of this issue, comparing a variety of different neural network architectures for operator approximation across a range of problems arising from PDE models in continuum mechanics.

Figures (19)

• Figure 1: Schematic of neural network architectures in numerical experiments. Circles represent layers; the width of each layer is given in the circle. Edges represent transformations between layers; the type of transformation between each layer is noted above each edge. Nonlinear and linear transformations are standard fully-connected layers; the lift and project layers are defined in \ref{['eq: FNO lift downsample defs']}; the Fourier Neural Layer (FNL) is defined in \ref{['eq: FNL definition']}.
• Figure 2: Navier-Stokes problem: sample input and output functions (left and right, respectively).
• Figure 3: Navier-Stokes test problem: learned model vorticity predictions for inputs resulting in median (left) and largest (right) test errors for networks of size $w = 128$ / $d_f = 16$ trained on $N = 10000$ data.
• Figure 4: Helmholtz test problem: Left: schematic of unit domain with labeled boundaries. Center: Sample input wave speed field. Right: Sample output disturbance field.
• Figure 5: Helmholtz test problem: learned model predictions for inputs resulting in median (left) and largest (right) test errors for networks of size $w = 128$ / $d_f = 16$ trained on $N = 10000$ data.

Theorems & Definitions (5)

• Remark 2.1
• Remark 2.2
• Remark 3.1
• Remark 3.2
• Remark 3.3