Diffusion Image Prior

TL;DR

This work tackles blind image restoration under unknown or complex degradations by introducing the Diffusion Image Prior (DIIP), a training-free approach that fixes a pretrained diffusion model and optimizes its input to align with degraded observations. The method leverages the diffusion model's strong implicit prior and uses self-supervised early stopping to prevent overfitting, enabling restoration across a broad range of degradations without explicit degradation models. DIIP demonstrates state-of-the-art performance on CelebA and ImageNet for tasks including denoising, super-resolution, JPEG artifact removal, waterdrop removal, and non-uniform blur, while maintaining reasonable efficiency. By revealing and exploiting biases of pretrained diffusion priors, the paper offers a practical, degradation-agnostic tool for real-world IR, reducing reliance on large, degradation-specific datasets.

Abstract

Zero-shot image restoration (IR) methods based on pretrained diffusion models have recently achieved significant success. These methods typically require at least a parametric form of the degradation model. However, in real-world scenarios, the degradation may be too complex to define explicitly. To handle this general case, we introduce the Diffusion Image Prior (DIIP). We take inspiration from the Deep Image Prior (DIP)[16], since it can be used to remove artifacts without the need for an explicit degradation model. However, in contrast to DIP, we find that pretrained diffusion models offer a much stronger prior, despite being trained without knowledge from corrupted data. We show that, the optimization process in DIIP first reconstructs a clean version of the image before eventually overfitting to the degraded input, but it does so for a broader range of degradations than DIP. In light of this result, we propose a blind image restoration (IR) method based on early stopping, which does not require prior knowledge of the degradation model. We validate DIIP on various degradation-blind IR tasks, including JPEG artifact removal, waterdrop removal, denoising and super-resolution with state-of-the-art results.

Figures (8)

• Figure 1: In practical applications, image degradation is often unknown, intricate, and challenging to model, such as when removing JPEG artifacts. In comparison to the recent state-of-the-art DreamClean xiaodreamclean, DIffusion Image Prior (DIIP) consistently produces results that more effectively preserve the original identity of the image.
• Figure 2: We demonstrate DIIP on several degradation-blind image restoration tasks. Top: Degraded input. Middle: Our prediction. Bottom: Ground truth.
• Figure 3: Intermediate outputs of the iterative reconstruction of a degraded image in the case of DIP (top) and our proposed optimization scheme using a frozen pretrained diffusion model (bottom). In the case of a pre-trained diffusion model, the optimization consistently produces clean images at an intermediate stage irrespective of the type of degradation ((a) noise or (b) blur).
• Figure 4: Stopping criterion in the case of degradations that introduce high-frequency artifacts: We study the optimization trend across varying levels of degradation that introduce high-frequency artifacts, focusing specifically on input images with different noise levels. (a) Shows the normalized slopes of the loss functions on input images with increasing noise levels. (b) Shows the Peak Signal-to-Noise Ratio (PSNR) relative to the clean reference image as the optimization progresses for various noise intensities. We observe that the slopes in (a) reach their minima in correspondence to iterations where the corresponding PSNR in (b) is maximal. For noise-based degradations (which introduce high-frequency artifacts), the slope in (a) can then serve as an indicator of peak performance, providing guidance on the optimal stopping point for the optimization process.
• Figure 5: Stopping criterion for degradations that remove high-frequency details from the image. Here we show the trend of the variance of the Laplacian operator throughout our optimization scheme. After the first regime, in case of blurry image the variance of the Laplacian keeps decreasing while in the case of a sharp image the optimization adds more high frequency details and the variance of the Laplacian does not decrease. A similar trend to the clean image is observed for a noisy image, which shows that a separate stopping criterion is needed.