Linearly Constrained Diffusion Implicit Models

TL;DR

CDIM introduces a fast, theoretically grounded framework for solving noisy linear inverse problems with pretrained diffusion priors. By constraining projection steps to stay within a plausible forward-process residual region, CDIM dramatically reduces the number of required projections while ensuring measurement consistency, and it even achieves exact noiseless constraint satisfaction. The method extends to Poisson noise via Pearson residuals and demonstrates strong performance across super-resolution, inpainting, deblurring, and 3D reprojection, with substantial speedups over prior approaches. Overall, CDIM advances the speed-quality Pareto frontier for linear inverse problems using diffusion models and adaptive projection strategies.

Abstract

We introduce Linearly Constrained Diffusion Implicit Models (CDIM), a fast and accurate approach to solving noisy linear inverse problems using diffusion models. Traditional diffusion-based inverse methods rely on numerous projection steps to enforce measurement consistency in addition to unconditional denoising steps. CDIM achieves a 10-50x reduction in projection steps by dynamically adjusting the number and size of projection steps to align a residual measurement energy with its theoretical distribution under the forward diffusion process. This adaptive alignment preserves measurement consistency while substantially accelerating constrained inference. For noise-free linear inverse problems, CDIM exactly satisfies the measurement constraints with few projection steps, even when existing methods fail. We demonstrate CDIM's effectiveness across a range of applications, including super-resolution, denoising, inpainting, deblurring, and 3D point cloud reprojection. Code and an interactive demo can be found on our project website.

Figures (19)

• Figure 1: We show several applications of our method including image colorization, denoising, inpainting, and sparse recovery. We highlight the fact that we can handle general noise distributions, such as Poisson noise, and that our method runs in as little as 3 seconds.
• Figure 2: The family of CDIM methods (top left corner) simultaneously achieves strong generation strong quality and extremely fast inference compared to other inverse solvers. We plot the inference speed and average LPIPS image quality score (inverted) averaged across multiple inverse tasks on the FFHQ dataset. "Ours" uses $T'=50$ denoising steps while "Ours fast" uses $T'=25$ denoising steps
• Figure 3: Results on a 50% noisy inpainting task with $\sigma_y = 0.2$. (a) is the noisy partial observation. (b) is generated by CDIM (Algorithm \ref{['alg:cdim']}) without considering the Gaussian measurement noise, showing that we can exactly match the constraint even when the observation is out of distribution. (c) is generated by CDIM and using the values from Appendix \ref{['app:noisy_Rt']} that consider the Gaussian measurement noise.
• Figure 4: We show the conceptual overview of CDIM for a 50% inpainting task without measurement noise. (Left) We compute the Tweedie posterior estimate $\hat{{\mathbf{x}}}_0({\mathbf{x}}_t)$ then apply the linear operator ${\mathbf{A}}$. This value ${\mathbf{A}}\hat{{\mathbf{x}}}_0({\mathbf{x}}_t)$ is compared with the observation ${\mathbf{y}}$ to obtain our loss $L_t = \|{\mathbf{A}}\hat{{\mathbf{x}}}_0({\mathbf{x}}_{t}) - {\mathbf{y}}\|^2$. We then update our iterate ${\mathbf{x}}_t$ with steps on $\nabla_{{\mathbf{x}}_t}L_t$. (Right) Our proxy residual $R_t = \|{\mathbf{A}}{\mathbf{x}}_t - {\mathbf{y}}\|^2$ has an anlytical $\chi^2$ distribution under the forward noising process and is used to guide the step size and number of steps for the left side process.
• Figure 5: We show the results of stopping at various points within the plausible region defined by $\mu_R({\mathbf{y}}) + c\cdot\sigma_R({\mathbf{y}})$ for different values of $c$. For $c<1$ the results do not change dramatically, but numerically still improve (see Appendix \ref{['app:ablation_studies']}).

Theorems & Definitions (1)

• Proposition 1