Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score-based Diffusion Models in Function Space

Jae Hyun Lim, Nikola B. Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, Christopher Pal, Arash Vahdat, Anima Anandkumar

TL;DR

This work extends diffusion models to function spaces by introducing Denoising Diffusion Operators (DDOs) that operate with function-valued Gaussian noise and score learning in Hilbert spaces. It develops a rigorous infinite-dimensional score framework, leveraging Cameron–Martin spaces and Radon–Nikodym densities, and demonstrates discretization-invariant sampling via Langevin dynamics learned with neural operators. The approach is validated on diverse function-valued tasks (Navier–Stokes, InSAR, MNIST-SDF, Darcy flow), showing favorable accuracy, stability across resolutions, and the ability to perform super-resolution without retraining. The work also provides theoretical results on measure equivalence, approximation by neural operators, and conditional sampling, outlining clear paths for extending diffusion modeling to complex function-valued data and inverse problems.

Abstract

Diffusion models have recently emerged as a powerful framework for generative modeling. They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising. Despite their tremendous success, they are mostly formulated on finite-dimensional spaces, e.g., Euclidean, limiting their applications to many domains where the data has a functional form, such as in scientific computing and 3D geometric data analysis. This work introduces a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. In DDOs, the forward process perturbs input functions gradually using a Gaussian process. The generative process is formulated by a function-valued annealed Langevin dynamic. Our approach requires an appropriate notion of the score for the perturbed data distribution, which we obtain by generalizing denoising score matching to function spaces that can be infinite-dimensional. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution. We theoretically and numerically verify the applicability of our approach on a set of function-valued problems, including generating solutions to the Navier-Stokes equation viewed as the push-forward distribution of forcings from a Gaussian Random Field (GRF), as well as volcano InSAR and MNIST-SDF.

Score-based Diffusion Models in Function Space

TL;DR

This work extends diffusion models to function spaces by introducing Denoising Diffusion Operators (DDOs) that operate with function-valued Gaussian noise and score learning in Hilbert spaces. It develops a rigorous infinite-dimensional score framework, leveraging Cameron–Martin spaces and Radon–Nikodym densities, and demonstrates discretization-invariant sampling via Langevin dynamics learned with neural operators. The approach is validated on diverse function-valued tasks (Navier–Stokes, InSAR, MNIST-SDF, Darcy flow), showing favorable accuracy, stability across resolutions, and the ability to perform super-resolution without retraining. The work also provides theoretical results on measure equivalence, approximation by neural operators, and conditional sampling, outlining clear paths for extending diffusion modeling to complex function-valued data and inverse problems.

Abstract

Diffusion models have recently emerged as a powerful framework for generative modeling. They consist of a forward process that perturbs input data with Gaussian white noise and a reverse process that learns a score function to generate samples by denoising. Despite their tremendous success, they are mostly formulated on finite-dimensional spaces, e.g., Euclidean, limiting their applications to many domains where the data has a functional form, such as in scientific computing and 3D geometric data analysis. This work introduces a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. In DDOs, the forward process perturbs input functions gradually using a Gaussian process. The generative process is formulated by a function-valued annealed Langevin dynamic. Our approach requires an appropriate notion of the score for the perturbed data distribution, which we obtain by generalizing denoising score matching to function spaces that can be infinite-dimensional. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution. We theoretically and numerically verify the applicability of our approach on a set of function-valued problems, including generating solutions to the Navier-Stokes equation viewed as the push-forward distribution of forcings from a Gaussian Random Field (GRF), as well as volcano InSAR and MNIST-SDF.
Paper Structure (55 sections, 13 theorems, 106 equations, 56 figures, 5 tables, 1 algorithm)

This paper contains 55 sections, 13 theorems, 106 equations, 56 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background: Denoising Score Matching in Finite Dimensions
  4. Denoising Diffusion Operators (DDO)
  5. Denoising score matching on function spaces
  6. Smoothing Operators
  7. Approximation Theory
  8. Langevin Dynamics
  9. Multiple Noise Scales and Annealed Langevin Dynamics
  10. Conditional Sampling
  11. Numerical Experiments
  12. Gaussian Mixture
  13. Navier-Stokes
  14. Volcano Dataset
  15. Results
  16. ...and 40 more sections

Key Result

Theorem 1

The perturbed measure $\nu$ and the centered Gaussian $\mu_0$ are equivalent in the sense of measures, which we denote by $\nu \sim \mu_0$.

Figures (56)

  • Figure 1: Overview of our approach. While in the finite-dimensional case, inputs are discretized, we work directly in function space, on continuous inputs, here 1D functions on $\mathbb{{R}}$. Noise is first added to the training samples during the forward process. A Neural Operator is used to estimate a score operator (Sec. \ref{['subsec:cond_score_matching']}) by minimizing the simplified loss in Eq. Equation \ref{['eq:conditional_score_matching_simple']}. Samples are generated using Langevin dynamics (Sec. \ref{['subsec:lagevin_dynamics']}). Using structured noise enables efficient learning in function space while white noise does not as the model capacity required grows with the resolution.
  • Figure 4: Results of Volcano dataset experiments (Section \ref{['subsec:volcano']}): Samples from the best performing FNO diffusion model with an RBF scale of $\gamma = 0.05$. For both histograms, $M = 1024$ generated samples were used to compute skew and variance.
  • Figure 7: Posterior Sample Statistics (Section \ref{['subsec:darcyflow']}): The sample (a) mean and (b) variance of posterior samples of the MCMC as well as the learned GANO, MultilevelDiff, and DDO models at various resolutions. 10,000 samples are used.
  • Figure 8: Posterior Samples at 64$\times$64 resolution (Section \ref{['subsec:darcyflow']}): The samples of the MCMC as well as the learned GANO, MultilevelDiff, and DDO models at the training resolution (64$\times$64).
  • Figure 9: Noise Regularity Test error when training with two different loss functions across different resolutions. Red curve is re-scaled so that it matches the error of the blue curve at the lowest resolution for the sake of visualization.
  • ...and 51 more figures

Theorems & Definitions (17)

  • Theorem 1: Measure Equivalence
  • Example 1
  • Theorem 2: Denoising Score Matching
  • Theorem 3: Score Approximation
  • Remark 4
  • Theorem 5
  • Theorem 6
  • Corollary 7
  • Lemma 8
  • Theorem 9
  • ...and 7 more