A Theoretical Justification for Image Inpainting using Denoising Diffusion Probabilistic Models

TL;DR

The work addresses image inpainting with diffusion models by assuming data lie on a low-dimensional subspace and proving that diffusion priors can recover missing regions without retraining. It reveals a misalignment bias in RePaint and proposes RePaint$^+$, which realigns drift and dispersion to achieve linear convergence toward the true sample, with provable recovery in the two-state setting and extensions to noisy generators. The analysis provides a universal-mask principle showing inpainting generalizes across unseen masks, derives a closed-form generative solution for two-state diffusion, and demonstrates that resampling intermediate states yields stronger, faster convergence than merely slowing diffusion. Collectively, these results offer theoretical guarantees and practical guidance for diffusion-based inpainting, including mask handling and resampling strategies, with potential extensions to nonlinear manifolds.

Abstract

We provide a theoretical justification for sample recovery using diffusion based image inpainting in a linear model setting. While most inpainting algorithms require retraining with each new mask, we prove that diffusion based inpainting generalizes well to unseen masks without retraining. We analyze a recently proposed popular diffusion based inpainting algorithm called RePaint (Lugmayr et al., 2022), and show that it has a bias due to misalignment that hampers sample recovery even in a two-state diffusion process. Motivated by our analysis, we propose a modified RePaint algorithm we call RePaint$^+$ that provably recovers the underlying true sample and enjoys a linear rate of convergence. It achieves this by rectifying the misalignment error present in drift and dispersion of the reverse process. To the best of our knowledge, this is the first linear convergence result for a diffusion based image inpainting algorithm.

Figures (5)

• Figure 1: Example demonstrating the bias in RePaint lugmayr2022repaint. Starting from the Gaussian prior $\overleftarrow{{\bm{x}}_1}$ (green circles), reverse SDE as proposed by RePaint generates $\overleftarrow{{\bm{x}}_0}$ (brown circle) that matches with the true data $\overrightarrow{{\bm{x}}_0}$ (blue square) in known coordinates, but differs in the inpainted region. In the figure, the blue squares along the $\sim 56^\circ$ line represents the true samples, whereas the brown circles along the $\sim 10^\circ$ line represents the samples recovered by RePaint. Note that the recovered samples match the true samples along the x-coordinate (known data), but have a bias along the y-coordinate (missing data).
• Figure 2: See caption in the published version.
• Figure 3: The data generating distribution is supported on a linear manifold with Gaussian marginals.
• Figure 4: Visualization of the forward and reverse processes in DDPM. The forward SDE pushes samples drawn from the data manifold towards the support of Gaussian prior. The reverse SDE discovers the subspace underneath data distribution.
• Figure 5: Running reverse SDE on top of the inpainted samples generated by RePaint. The final reverse step is highlighted by red arrows ($\rightarrow$). It fails to recover the true underlying sample.

Theorems & Definitions (16)

• Theorem 3: Generative Modeling
• proof
• Theorem 4: Image Inpainting
• proof
• Theorem 6
• proof
• Corollary 7
• proof
• Theorem 8
• proof