Reconciling Stochastic and Deterministic Strategies for Zero-shot Image Restoration using Diffusion Model in Dual

TL;DR

Zero-shot image restoration faces a fidelity-perception trade-off and cross-task generalization challenges. The authors propose Reconciling Diffusion Model in Dual (RDMD), a unified HQS-like framework that derives two complementary regularizers from a single pre-trained diffusion prior to perform both stochastic sampling and deterministic regression, regulated by a single parameter $\tau$. This approach enables explicit control of the distortion-perception tradeoff and reduces model complexity compared to dual-network priors. Across Gaussian/Motion deblurring and super-resolution tasks on FFHQ and ImageNet, RDMD achieves superior or competitive results, highlighting its practicality for diverse IR problems without task-specific retraining.

Abstract

Plug-and-play (PnP) methods offer an iterative strategy for solving image restoration (IR) problems in a zero-shot manner, using a learned \textit{discriminative denoiser} as the implicit prior. More recently, a sampling-based variant of this approach, which utilizes a pre-trained \textit{generative diffusion model}, has gained great popularity for solving IR problems through stochastic sampling. The IR results using PnP with a pre-trained diffusion model demonstrate distinct advantages compared to those using discriminative denoisers, \ie improved perceptual quality while sacrificing the data fidelity. The unsatisfactory results are due to the lack of integration of these strategies in the IR tasks. In this work, we propose a novel zero-shot IR scheme, dubbed Reconciling Diffusion Model in Dual (RDMD), which leverages only a \textbf{single} pre-trained diffusion model to construct \textbf{two} complementary regularizers. Specifically, the diffusion model in RDMD will iteratively perform deterministic denoising and stochastic sampling, aiming to achieve high-fidelity image restoration with appealing perceptual quality. RDMD also allows users to customize the distortion-perception tradeoff with a single hyperparameter, enhancing the adaptability of the restoration process in different practical scenarios. Extensive experiments on several IR tasks demonstrate that our proposed method could achieve superior results compared to existing approaches on both the FFHQ and ImageNet datasets.

Figures (6)

• Figure 1: An example of super-resolution (4$\times$) with noise level 0.05. From (a) to (e): (a) low-resolution input, (b) original image, (c) ours result via pure deterministic regression, (d) ours result via pure stochastic sampling, (e) ours by reconciling these two variants, respectively.
• Figure 2: Comparison of restoration processes: deterministic regression, stochastic sampling, and our proposed reconciliation approach. The deterministic regression method (top left) generates high-fidelity restorations through pointwise estimations. The stochastic sampling method (bottom left) produces diverse outputs by sampling from a learned distribution, enhancing perceptual quality. Our reconciliation approach (right) unifies deterministic estimation for accurate details with stochastic estimation for diverse possibilities, achieving both high fidelity and perceptual richness in the final restored images.
• Figure 3: An example of controlling distortion-perception tradeoff for 4$\times$ super-resolution via adjusting different values of $\tau$. Specifically, $\tau \to 0$ leads to deterministic regression while $\tau \to 1$ promotes the stochasticity.
• Figure 4: Example of IR results of (a) MCG chung2022improving, (b) DPS chung2023diffusion, (c) Resample song2024solving, (d) BIRD chihaoui2024blind, (e) RED romano2017little, (f) DiffPIR zhu2023denoising, and Ours on FFHQ and ImageNet datasets.
• Figure 5: Results of SR (4$\times$) on FFHQ dataset using different iteration step $T$.