Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the role of memorization in learned priors for geophysical inverse problems

Ali Siahkoohi, Davide Sabeddu

Abstract

Learned priors based on deep generative models offer data-driven regularization for seismic inversion, but training them requires a dataset of representative subsurface models -- a resource that is inherently scarce in geoscience applications. Since the training objective of most generative models can be cast as maximum likelihood on a finite dataset, any such model risks converging to the empirical distribution -- effectively memorizing the training examples rather than learning the underlying geological distribution. We show that the posterior under such a memorized prior reduces to a reweighted empirical distribution -- i.e., a likelihood-weighted lookup among the stored training examples. For diffusion models specifically, memorization yields a Gaussian mixture prior in closed form, and linearizing the forward operator around each training example gives a Gaussian mixture posterior whose components have widths and shifts governed by the local Jacobian. We validate these predictions on a stylized inverse problem and demonstrate the consequences of memorization through diffusion posterior sampling for full waveform inversion.

On the role of memorization in learned priors for geophysical inverse problems

Abstract

Learned priors based on deep generative models offer data-driven regularization for seismic inversion, but training them requires a dataset of representative subsurface models -- a resource that is inherently scarce in geoscience applications. Since the training objective of most generative models can be cast as maximum likelihood on a finite dataset, any such model risks converging to the empirical distribution -- effectively memorizing the training examples rather than learning the underlying geological distribution. We show that the posterior under such a memorized prior reduces to a reweighted empirical distribution -- i.e., a likelihood-weighted lookup among the stored training examples. For diffusion models specifically, memorization yields a Gaussian mixture prior in closed form, and linearizing the forward operator around each training example gives a Gaussian mixture posterior whose components have widths and shifts governed by the local Jacobian. We validate these predictions on a stylized inverse problem and demonstrate the consequences of memorization through diffusion posterior sampling for full waveform inversion.
Paper Structure (9 sections, 5 equations, 5 figures, 1 table)

This paper contains 9 sections, 5 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Inverse problems with learned priors
  3. The empirical prior and the lookup-table posterior
  4. Memorized diffusion models as Gaussian mixture priors
  5. Posterior inference under the memorized diffusion prior
  6. Numerical illustration
  7. Stylized example: memorized diffusion model
  8. Frequency-domain FWI with learned diffusion priors
  9. Discussion and conclusions

Figures (5)

  • Figure 1: Stylized posterior collapse. Top: 1D posterior (red shaded) and linearized approximation (black dashed, Equation \ref{['eq:linearized-posterior']}) at $\sigma = 0.5$, $0.3$, and $0.05$. Bottom: 2D linearized posterior (black contours) overlaid on exact posterior (red shades). As $\sigma$ decreases, the posterior collapses from a smooth, data-informed distribution to discrete spikes at training examples (cf. Equation \ref{['eq:lookup-table']}).
  • Figure 2: (a) True velocity model used for the Helmholtz FWI experiment. (b) Data misfit during the reverse diffusion process for $N\!=\!50$, $200$, and $1000$.
  • Figure 3: Most memorized $N\!=\!50$ samples and their nearest training neighbors. Left pair: unconditional diffusion samples are nearly indistinguishable from training data ($r \approx 0.01$). Right pair: DPS posterior samples retain memorized structure but the likelihood introduces perturbations ($r \approx 0.2$--$0.34$).
  • Figure 4: DPS posterior analysis ($256$ samples, $N\!=\!50$, $200$, $1000$). (a--c) Posterior mean. (d--f) Pointwise standard deviation. (g--i) First two KL coefficients: posterior samples (red), training data (blue), true model (star).
  • Figure 5: Calibration: pointwise error vs. posterior std in KL space.