Blind Image Restoration via Fast Diffusion Inversion

TL;DR

This work proposes Blind Image Restoration via fast Diffusion inversion (BIRD) a blind IR method that jointly optimizes for the degradation model parameters and the restored image to mitigate the computational cost associated with inverting a fully unrolled diffusion model.

Abstract

Image Restoration (IR) methods based on a pre-trained diffusion model have demonstrated state-of-the-art performance. However, they have two fundamental limitations: 1) they often assume that the degradation operator is completely known and 2) they alter the diffusion sampling process, which may result in restored images that do not lie onto the data manifold. To address these issues, we propose Blind Image Restoration via fast Diffusion inversion (BIRD) a blind IR method that jointly optimizes for the degradation model parameters and the restored image. To ensure that the restored images lie onto the data manifold, we propose a novel sampling technique on a pre-trained diffusion model. A key idea in our method is not to modify the reverse sampling, i.e, not to alter all the intermediate latents, once an initial noise is sampled. This is ultimately equivalent to casting the IR task as an optimization problem in the space of the input noise. Moreover, to mitigate the computational cost associated with inverting a fully unrolled diffusion model, we leverage the inherent capability of these models to skip ahead in the forward diffusion process using large time steps. We experimentally validate BIRD on several image restoration tasks and show that it achieves state of the art performance on all of them. Our code is available at https://github.com/hamadichihaoui/BIRD.

Figures (10)

• Figure 1: Blind image deblurring with unknown motion blur. (a): blurry input image. From (b) to (e): The top row shows the predictions of BlindDPS chung2023parallel as the iterations increase; the bottom row shows the corresponding estimated blur kernel. From (f) to (i): Same estimates as in (b) to (e), but obtained from BIRD, our proposed method. (j) is the ground truth sharp image (top) and ground truth blur kernel (bottom). Notice that BlindDPS chung2023parallel trains a score-based model for the kernel estimation, while BIRD does not use any training and can adapt to any new kernel directly at test time. BIRD yields always natural images at every iteration of the reconstruction procedure. Finally, notice that despite recovering a suboptimal blur kernel, the image reconstructed with BIRD is more similar to the ground truth image than with BlindDPS thanks to the robustness of our image generation procedure.
• Figure 2: We demonstrate BIRD on several blind image restoration problems (i.e., when the values of the degradation model are unknown): Gaussian deblurring, motion deblurring, superresolution (SR) (it includes additional Gaussian blur) and denoising with an unknown noise distribution. BIRD is applicable to a single degraded image and does not require re-training or fine-tuning of the prior model (we use a diffusion model). Although some of the generated degraded images use Gaussian blur, BIRD recovers a generic blur kernel (without making any Gaussianity assumption).
• Figure 3: Illustration of our proposed accelerated image sampling of the pre-trained DDIM. (a) intial noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ ($T=1000$). (b), (c), (d), (e) and (f) are samples $x_0$ generated using $\text{DDIMReverse}(x_T, \delta t)$ with $\delta t = 100, 50, 20, 1$ respectively. Notice how the generated images are all realistic regardless of the choice of the step size $\delta t$.
• Figure 4: Reconstruction results using BIRD on samples from CelebA and ImageNet validation datasets. The PSNR mean and standard deviation are computed over 10 runs.
• Figure 5: Qualitative comparisons of $4\times$ SR on ImageNet. Each column shows two examples. From left to right: input low-resolution images, GDP fei2023generative, BlindDPS chung2023parallel, BIRD (our method), and the ground truth (GT) high-resolution and noise-free image.