Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performance

Jing Jia, Wei Yuan, Sifan Liu, Liyue Shen, Guanyang Wang

TL;DR

The paper investigates when diffusion priors used for inverse problems remain effective even if they are weak (low-fidelity or mismatched). It combines empirical benchmarks across tasks with a Bayesian-consistency theory showing that highly informative measurements (high-dimensional observations) can overwhelm prior differences, causing the posterior to concentrate near the true signal. The authors introduce an initial-noise optimization approach with AdamSphere and HoldoutTopK to stabilize reconstructions under weak priors and demonstrate strong cross-domain performance, including image-domain and latent-diffusion settings. The findings justify using weak priors as a practical default in data-rich regimes and identify clear failure modes, guiding future work on hybrid algorithms and sharper identifiability thresholds.

Abstract

Can a diffusion model trained on bedrooms recover human faces? Diffusion models are widely used as priors for inverse problems, but standard approaches usually assume a high-fidelity model trained on data that closely match the unknown signal. In practice, one often must use a mismatched or low-fidelity diffusion prior. Surprisingly, these weak priors often perform nearly as well as full-strength, in-domain baselines. We study when and why inverse solvers are robust to weak diffusion priors. Through extensive experiments, we find that weak priors succeed when measurements are highly informative (e.g., many observed pixels), and we identify regimes where they fail. Our theory, based on Bayesian consistency, gives conditions under which high-dimensional measurements make the posterior concentrate near the true signal. These results provide a principled justification on when weak diffusion priors can be used reliably.

Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performance

TL;DR

The paper investigates when diffusion priors used for inverse problems remain effective even if they are weak (low-fidelity or mismatched). It combines empirical benchmarks across tasks with a Bayesian-consistency theory showing that highly informative measurements (high-dimensional observations) can overwhelm prior differences, causing the posterior to concentrate near the true signal. The authors introduce an initial-noise optimization approach with AdamSphere and HoldoutTopK to stabilize reconstructions under weak priors and demonstrate strong cross-domain performance, including image-domain and latent-diffusion settings. The findings justify using weak priors as a practical default in data-rich regimes and identify clear failure modes, guiding future work on hybrid algorithms and sharper identifiability thresholds.

Abstract

Can a diffusion model trained on bedrooms recover human faces? Diffusion models are widely used as priors for inverse problems, but standard approaches usually assume a high-fidelity model trained on data that closely match the unknown signal. In practice, one often must use a mismatched or low-fidelity diffusion prior. Surprisingly, these weak priors often perform nearly as well as full-strength, in-domain baselines. We study when and why inverse solvers are robust to weak diffusion priors. Through extensive experiments, we find that weak priors succeed when measurements are highly informative (e.g., many observed pixels), and we identify regimes where they fail. Our theory, based on Bayesian consistency, gives conditions under which high-dimensional measurements make the posterior concentrate near the true signal. These results provide a principled justification on when weak diffusion priors can be used reliably.
Paper Structure (38 sections, 6 theorems, 58 equations, 15 figures, 10 tables)

This paper contains 38 sections, 6 theorems, 58 equations, 15 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Weak diffusion generators as priors
  3. Setup and our algorithm
  4. Main observations:
  5. When data dominates the prior
  6. Posterior consistency under i.i.d. data
  7. High dimensional inverse problem under generative prior
  8. Score and MSE.
  9. Experiments
  10. Cross-Domain Inverse Problem Solving
  11. Comparison with Optimization-Based Methods and Ablations
  12. Failure Modes
  13. Latent Diffusion Applications on ImageNet
  14. Conclusion and Discussion
  15. Additional theoretical results and proofs
  16. ...and 23 more sections

Key Result

Theorem 3.1

Let $\{P_x\}_{x\in\mathcal{X}}$ be a statistical model and let $Y_1,Y_2,\ldots$ be i.i.d. from $P_{x^\star}$. Under standard regularity and identifiability assumptions, and assuming the prior $\pi$ puts positive mass in every neighborhood of $x^\star$, the posterior $\pi(\cdot\mid y_{1:N})$ concentr

Figures (15)

  • Figure 1: Image reconstruction using only a 3-step DDIM generator as the prior: matched prior (top) vs. mismatched prior (bottom). Diffusion models trained on bedroom images (resp. face images) can reconstruct face images (resp. bedroom images), shown in the bottom-left (resp. bottom-right) panels. From left to right, we display intermediate reconstructions over optimization iterations. The Reference column shows the clean image, and the Measurement column shows the noisy observation.
  • Figure 2: Posterior concentrates around $x^\star$ as $N$ grows.
  • Figure 3: Visual comparison on box inpainting and super-resolution tasks. Top row: box inpainting with a $0.6 \times 0.6$ mask. Bottom row: $16\times$ super-resolution.
  • Figure 4: Difference in LPIPS between our method and the DPS baseline for different mask sizes. The dashed horizontal line denotes equal performance with DPS.
  • Figure 5: Compute-matched comparison between DPS and initial-noise optimization methods.
  • ...and 10 more figures

Theorems & Definitions (10)

  • Theorem 3.1: Posterior consistency, informal
  • Theorem 3.2
  • Theorem A.1: Posterior consistency
  • Proposition A.2
  • proof
  • Theorem A.3: Posterior collapse to the best selection-score mode
  • Remark A.4
  • proof
  • Proposition A.5: Identifiability for random inpainting
  • proof