Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Infinite Neural Operators: Gaussian processes on functions

Daniel Augusto de Souza, Yuchen Zhu, Harry Jake Cunningham, Yuri Saporito, Diego Mesquita, Marc Peter Deisenroth

TL;DR

This work proves that infinitely wide neural operators converge to function-valued Gaussian processes, establishing a principled probabilistic framework for operator learning. It derives closed-form operator-valued covariances for two NO parametrizations, including the Fourier neural operator and a toroidal Matérn kernel, enabling GP posteriors for operator regression such as PDE mappings. The results provide a path to quantify uncertainty and incorporate inductive biases via kernel-based operator learning, with empirical validation showing alignment at initialization and competitive regression behavior. Overall, the paper bridges infinite-width theory and practical kernel-based operator learning, offering a foundation for Bayesian NO architectures and future extensions to other NO designs.

Abstract

A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and transformers) induce Gaussian process (GP) priors over their outputs. These relationships provide both an accurate characterization of the prior predictive distribution and enable the use of GP machinery to improve the uncertainty quantification of deep neural networks. In this work, we extend this connection to neural operators (NOs), a class of models designed to learn mappings between function spaces. Specifically, we show conditions for when arbitrary-depth NOs with Gaussian-distributed convolution kernels converge to function-valued GPs. Based on this result, we show how to compute the covariance functions of these NO-GPs for two NO parametrizations, including the popular Fourier neural operator (FNO). With this, we compute the posteriors of these GPs in regression scenarios, including PDE solution operators. This work is an important step towards uncovering the inductive biases of current FNO architectures and opens a path to incorporate novel inductive biases for use in kernel-based operator learning methods.

Infinite Neural Operators: Gaussian processes on functions

TL;DR

This work proves that infinitely wide neural operators converge to function-valued Gaussian processes, establishing a principled probabilistic framework for operator learning. It derives closed-form operator-valued covariances for two NO parametrizations, including the Fourier neural operator and a toroidal Matérn kernel, enabling GP posteriors for operator regression such as PDE mappings. The results provide a path to quantify uncertainty and incorporate inductive biases via kernel-based operator learning, with empirical validation showing alignment at initialization and competitive regression behavior. Overall, the paper bridges infinite-width theory and practical kernel-based operator learning, offering a foundation for Bayesian NO architectures and future extensions to other NO designs.

Abstract

A variety of infinitely wide neural architectures (e.g., dense NNs, CNNs, and transformers) induce Gaussian process (GP) priors over their outputs. These relationships provide both an accurate characterization of the prior predictive distribution and enable the use of GP machinery to improve the uncertainty quantification of deep neural networks. In this work, we extend this connection to neural operators (NOs), a class of models designed to learn mappings between function spaces. Specifically, we show conditions for when arbitrary-depth NOs with Gaussian-distributed convolution kernels converge to function-valued GPs. Based on this result, we show how to compute the covariance functions of these NO-GPs for two NO parametrizations, including the popular Fourier neural operator (FNO). With this, we compute the posteriors of these GPs in regression scenarios, including PDE solution operators. This work is an important step towards uncovering the inductive biases of current FNO architectures and opens a path to incorporate novel inductive biases for use in kernel-based operator learning methods.
Paper Structure (38 sections, 3 theorems, 82 equations, 3 figures)

This paper contains 38 sections, 3 theorems, 82 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Operator learning and neural operators
  4. Probability in Hilbert spaces
  5. Operator valued kernels and Hilbert space valued Gaussian processes
  6. Infinite-width neural operators as Gaussian processes
  7. Operator-valued covariance functions
  8. Outline of the proof for \ref{['theorem:inf-no']}
  9. Parametrizations and computations
  10. Fourier neural operator
  11. Parameterization of an FNO.
  12. Toroidal Matérn operator
  13. Experimental validation
  14. Empirical demonstration of results
  15. Regression tasks
  16. ...and 23 more sections

Key Result

Theorem 2.1

Let $\set{H}$ be a separable Hilbert space and $\braces{x_j}_{j \in \mathbb{N}}$ be a countable sequence of identically distributed random elements. Consider the sample average $y_N = \parens{1/N}\sum_{j=1}^N x_j$. If, for any $j$, the expected norm $\E\bcall{\norm{x_j}}$ exists, then, the sequence

Figures (3)

  • Figure 1: A density estimation of the BBD5E81F77B4 empirical distribution of the output of increasing channel dimension compared to the FFFFFF000000 infinite width distribution. On top of each plot we show the total variation distance of the empirical distribution against the infinite width distribution.
  • Figure 2: Plot of the MC estimate for the variance of $\op{H}\bcall{\fun{f}}\call{0}$ against our analytical computation (Sec. \ref{['subsec:covariance-functions']}).
  • Figure 3: Results for the regression experiments. Mean and std. of test $\L{2}$ loss as a function of width $J$ for different band-limits $B$.

Theorems & Definitions (8)

  • Theorem 2.1: Strong law of large numbers Mourier1956
  • Theorem 3.1: Infinite-width neural operators are Gaussian processes
  • Lemma 3.2: Compositionality of covariance functions
  • Definition A.1: Matérn family of kernels on a closed manifold
  • proof
  • proof
  • Definition B.1: Iterated convergence in distribution
  • proof