Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Functional Regression with Neural Operator Flows

Yaozhong Shi, Angela F. Gao, Zachary E. Ross, Kamyar Azizzadenesheli

TL;DR

The notion of universal functional regression is introduced, in which the aim is to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression.

Abstract

Regression on function spaces is typically limited to models with Gaussian process priors. We introduce the notion of universal functional regression, in which we aim to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression. To do this, we develop Neural Operator Flows (OpFlow), an infinite-dimensional extension of normalizing flows. OpFlow is an invertible operator that maps the (potentially unknown) data function space into a Gaussian process, allowing for exact likelihood estimation of functional point evaluations. OpFlow enables robust and accurate uncertainty quantification via drawing posterior samples of the Gaussian process and subsequently mapping them into the data function space. We empirically study the performance of OpFlow on regression and generation tasks with data generated from Gaussian processes with known posterior forms and non-Gaussian processes, as well as real-world earthquake seismograms with an unknown closed-form distribution.

Universal Functional Regression with Neural Operator Flows

TL;DR

The notion of universal functional regression is introduced, in which the aim is to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression.

Abstract

Regression on function spaces is typically limited to models with Gaussian process priors. We introduce the notion of universal functional regression, in which we aim to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression. To do this, we develop Neural Operator Flows (OpFlow), an infinite-dimensional extension of normalizing flows. OpFlow is an invertible operator that maps the (potentially unknown) data function space into a Gaussian process, allowing for exact likelihood estimation of functional point evaluations. OpFlow enables robust and accurate uncertainty quantification via drawing posterior samples of the Gaussian process and subsequently mapping them into the data function space. We empirically study the performance of OpFlow on regression and generation tasks with data generated from Gaussian processes with known posterior forms and non-Gaussian processes, as well as real-world earthquake seismograms with an unknown closed-form distribution.
Paper Structure (24 sections, 17 equations, 25 figures, 3 tables, 3 algorithms)

This paper contains 24 sections, 17 equations, 25 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Neural Operator Flow
  5. Invertible Normalized Architecture
  6. Training
  7. Universal Functional Regression with OpFlow
  8. Experiments
  9. Universal Functional Regression
  10. Generation Tasks
  11. Conclusion
  12. Appendix
  13. Forward and Inverse processes of OpFlow
  14. Discretization agnostic property of OpFlow
  15. Parameters used in regression tasks
  16. ...and 9 more sections

Figures (25)

  • Figure 2: Model architecture of OpFlow, OpFlow is composed of a list of invertible operators. For the universal function regression task, OpFlow is the learnt prior, which is able to provide exact likehood estimation for function point evaluation. $\tilde{u}_{obs}$ is the noisy observation, $u_{\phi}$ is the posterior function of interest and $u_{\phi} = \mathcal{G}_{\theta}(a_\phi)$, where $\mathcal{G}_{\theta}$ is the learnt forward operator.
  • Figure 3: OpFlow regression on GP data. (a) Ground truth GP regression with observed data and predicted samples (b) OpFlow regression with observed data and predicted samples. (c) Uncertainty comparison between true GP and OpFlow predictions.
  • Figure 4: OpFlow regression on TGP data. (a) Ground truth TGP regression with observed data and predicted samples (b) OpFlow regression with observed data and predicted samples. (c) prior GP regression with observed data and predicted samples (c) Uncertainty comparison between true TGP and OpFlow predictions.
  • Figure 5: OpFlow regression on 32$\times$32 GRF data. (a) 32 random observations. (b) Predicted mean from OpFlow. (c) Ground truth mean from GP regression. (d) Misfit of the predicted mean. (e) Misfit of predicted uncertainty. (f) Predicted samples from OpFlow. (g) Predicted samples from GP regression.
  • Figure 6: OpFlow regression on seismic waveform data. (a) Ground truth waveform. (b) One predicted sample from OpFlow. (c) One predicted sample from the best fitted GP. (d) Predicted mean and uncertainty from OpFlow. (e) Predicted mean and uncertainty from the best fitted GP. (f) Samples from OpFlow. (g) Samples from the best fitted GP.
  • ...and 20 more figures