Deep Data Consistency: a Fast and Robust Diffusion Model-based Solver for Inverse Problems

TL;DR

This work tackles the slow, imperfect balance between data fidelity and realism in diffusion-model solvers for inverse problems. It introduces Deep Data Consistency (DDC), a neural residual updater trained with a variational bound objective to maximize the conditional posterior while minimally perturbing the diffusion process. DDC provides two components—DDC Sampling and DDC Variational Bound Training—that enable high-quality solutions with only 5 inference steps and ~0.77 seconds per image, across both linear and nonlinear tasks and multiple datasets using a single pre-trained model. Empirically, DDC outperforms or matches state-of-the-art solvers in fidelity and realism (PSNR, SSIM, LPIPS, FID) while offering substantial speedups and robustness to noise, underscoring its practical impact for diffusion-based inverse problem solving.

Abstract

Diffusion models have become a successful approach for solving various image inverse problems by providing a powerful diffusion prior. Many studies tried to combine the measurement into diffusion by score function replacement, matrix decomposition, or optimization algorithms, but it is hard to balance the data consistency and realness. The slow sampling speed is also a main obstacle to its wide application. To address the challenges, we propose Deep Data Consistency (DDC) to update the data consistency step with a deep learning model when solving inverse problems with diffusion models. By analyzing existing methods, the variational bound training objective is used to maximize the conditional posterior and reduce its impact on the diffusion process. In comparison with state-of-the-art methods in linear and non-linear tasks, DDC demonstrates its outstanding performance of both similarity and realness metrics in generating high-quality solutions with only 5 inference steps in 0.77 seconds on average. In addition, the robustness of DDC is well illustrated in the experiments across datasets, with large noise and the capacity to solve multiple tasks in only one pre-trained model.

Figures (16)

• Figure 1: DDC solves inverse problems in only 5 sampling steps. (left) Illustration of our proposed DDC method for a specific random inpainting task. It uses a neural network to update the data consistency step, providing an efficient way for image restoration. (right) Some representative results include Gaussian blur, super-resolution $\times8$, and JPEG restoration tasks with large Gaussian noise.
• Figure 2: The kurtosis of $\bm{\epsilon}_\theta (\bm{x}_t, t)$ when generating images in DPS chung2022diffusion, DDNM wang2022zero and unconditional generation. Fréchet Inception Distance heusel2017gans (FID) and structural similarity (SSIM) are reported to measure the realness and similarity.
• Figure 3: Representative results on linear inverse problems (a) with Gaussian noise $\sigma_y = 0.05$ and (b) without noise on ImageNet dataset. The degradations include SR $\times4$, SR $\times8$, Gaussian blur and random inpainting tasks.
• Figure 4: Representative results on linear inverse problems with Gaussian noise $\sigma_y = 0.05$ on CelebA dataset. The degradations include SR $\times4$ and random inpainting tasks.
• Figure 5: Quantitative results of the SR $\times4$ and random inpainting tasks on CelebA dataset with Gaussian noise strength $\sigma_y=0.05$. DDC networks are trained on ImageNet and CelebA datasets separately.