Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo

Idan Achituve, Hai Victor Habi, Amir Rosenfeld, Arnon Netzer, Idit Diamant, Ethan Fetaya

TL;DR

LD-SMC introduces a latent-space sequential Monte Carlo framework for inverse problems, leveraging auxiliary observations and a reverse-diffusion model to sample from pθ(z0|y0). By defining a latent generative model and crafting tailored targets, weights, and proposals, LD-SMC achieves asymptotically exact posterior sampling while guiding reconstructions through both data fidelity and natural image priors. Empirical results on ImageNet and FFHQ show LD-SMC delivering strong perceptual quality, especially in challenging inpainting tasks, outperforming several state-of-the-art baselines. The approach highlights a practical trade-off between computational cost and reconstruction fidelity, with clear pathways for optimization and extension to broader nonlinear corruption operators.

Abstract

In image processing, solving inverse problems is the task of finding plausible reconstructions of an image that was corrupted by some (usually known) degradation operator. Commonly, this process is done using a generative image model that can guide the reconstruction towards solutions that appear natural. The success of diffusion models over the last few years has made them a leading candidate for this task. However, the sequential nature of diffusion models makes this conditional sampling process challenging. Furthermore, since diffusion models are often defined in the latent space of an autoencoder, the encoder-decoder transformations introduce additional difficulties. To address these challenges, we suggest a novel sampling method based on sequential Monte Carlo (SMC) in the latent space of diffusion models. We name our method LD-SMC. We define a generative model for the data using additional auxiliary observations and perform posterior inference with SMC sampling based on a reverse diffusion process. Empirical evaluations on ImageNet and FFHQ show the benefits of LD-SMC over competing methods in various inverse problem tasks and especially in challenging inpainting tasks.

Inverse Problem Sampling in Latent Space Using Sequential Monte Carlo

TL;DR

LD-SMC introduces a latent-space sequential Monte Carlo framework for inverse problems, leveraging auxiliary observations and a reverse-diffusion model to sample from pθ(z0|y0). By defining a latent generative model and crafting tailored targets, weights, and proposals, LD-SMC achieves asymptotically exact posterior sampling while guiding reconstructions through both data fidelity and natural image priors. Empirical results on ImageNet and FFHQ show LD-SMC delivering strong perceptual quality, especially in challenging inpainting tasks, outperforming several state-of-the-art baselines. The approach highlights a practical trade-off between computational cost and reconstruction fidelity, with clear pathways for optimization and extension to broader nonlinear corruption operators.

Abstract

In image processing, solving inverse problems is the task of finding plausible reconstructions of an image that was corrupted by some (usually known) degradation operator. Commonly, this process is done using a generative image model that can guide the reconstruction towards solutions that appear natural. The success of diffusion models over the last few years has made them a leading candidate for this task. However, the sequential nature of diffusion models makes this conditional sampling process challenging. Furthermore, since diffusion models are often defined in the latent space of an autoencoder, the encoder-decoder transformations introduce additional difficulties. To address these challenges, we suggest a novel sampling method based on sequential Monte Carlo (SMC) in the latent space of diffusion models. We name our method LD-SMC. We define a generative model for the data using additional auxiliary observations and perform posterior inference with SMC sampling based on a reverse diffusion process. Empirical evaluations on ImageNet and FFHQ show the benefits of LD-SMC over competing methods in various inverse problem tasks and especially in challenging inpainting tasks.

Paper Structure

This paper contains 22 sections, 4 theorems, 16 equations, 10 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Method
  5. The Generative Model
  6. Sampling Procedure
  7. Initial Guess for ${\mathbf{z}}_0$
  8. Posterior Sampling
  9. Experiments
  10. Experimental Setting
  11. Experimental Results
  12. Analysis
  13. Conclusion
  14. Full Experimental Details
  15. Approximate Target Distributions
  16. ...and 7 more sections

Key Result

Theorem 4.1

(informal) Let ${\mathbb{P}}_N({\mathbf{z}}_{0:T}) = \sum_{i=1}^N w^{(i)}_0 \delta_{{\mathbf{z}}_{0:T}^{(i)}}({\mathbf{z}}_{0:T})$ be the discrete measure obtained by the function $\mathbf{SISR}$ in Algorithm algo:LD_SMC. Under regularity conditions ${\mathbb{P}}_N({\mathbf{z}}_{0:T})$ converges set

Figures (10)

  • Figure 1: LD-SMC solves inverse problem tasks in the latent space of autoencoders by utilizing auxiliary observations ${\mathbf{y}}_{1:T}$ initialized using the DDIM forward process. Then, sampling ${\mathbf{z}}_0$ from the posterior distribution is done based on the reverse diffusion process using sequential Monte Carlo. In the figure, ${\mathcal{D}}$ and ${\mathcal{A}}$ denote the decoder and the corruption operator respectively.
  • Figure 2: The graphical model of LD-SMC. In gray observed variables and in white are latent variables.
  • Figure 3: Comparison between LD-SMC and baseline methods on inpainting of ImageNet images.
  • Figure 4: Comparison between LD-SMC and baseline methods on Gaussian deblurring of ImageNet images.
  • Figure 5: FID and PSNR values when varying $s$ (left) and the number of particles $N$ (right) on ImageNet box (left) and free-form (right) inpainting tasks.
  • ...and 5 more figures

Theorems & Definitions (4)

  • Theorem 4.1
  • Theorem 3.1
  • Theorem 3.2
  • Proposition 3.3