Invert2Restore: Zero-Shot Degradation-Blind Image Restoration

TL;DR

This work introduces Invert2Restore, a zero-shot, training-free method that operates in both fully blind and partially blind settings -- requiring no prior knowledge of the degradation model or only partial knowledge of its parametric form without known parameters, and achieves high-fidelity results and generalizes well across various types of image degradation.

Abstract

Two of the main challenges of image restoration in real-world scenarios are the accurate characterization of an image prior and the precise modeling of the image degradation operator. Pre-trained diffusion models have been very successfully used as image priors in zero-shot image restoration methods. However, how to best handle the degradation operator is still an open problem. In real-world data, methods that rely on specific parametric assumptions about the degradation model often face limitations in their applicability. To address this, we introduce Invert2Restore, a zero-shot, training-free method that operates in both fully blind and partially blind settings -- requiring no prior knowledge of the degradation model or only partial knowledge of its parametric form without known parameters. Despite this, Invert2Restore achieves high-fidelity results and generalizes well across various types of image degradation. It leverages a pre-trained diffusion model as a deterministic mapping between normal samples and undistorted image samples. The key insight is that the input noise mapped by a diffusion model to a degraded image lies in a low-probability density region of the standard normal distribution. Thus, we can restore the degraded image by carefully guiding its input noise toward a higher-density region. We experimentally validate Invert2Restore across several image restoration tasks, demonstrating that it achieves state-of-the-art performance in scenarios where the degradation operator is either unknown or partially known.

Figures (8)

• Figure 1: We demonstrate Invert2Restore on several fully blind image restoration problems (i.e., when we do not know the parametric form of the degradation model): JPEG restoration (de-artifacting), deraining, raindrop removal, and also to partially-blind restoration problems (when only the parametric form of the degradation is known but its values are unknown): deblurring, superresolution (SR) . First row: input image. Second row: our prediction. Third row: ground-truth. Invert2Restore is applicable to a single degraded image and does not require re-training or fine-tuning of the prior model (in our case, a diffusion model).
• Figure 2: Our method vs DreamClean xiaodreamclean. Both methods can operate in full blindness regarding the degradation model. Left: DreamClean alters the diffusion reverse sampling. In contrast, in our method we only estimate and rectify the initial noise and do not alter the reverse sampling of the diffusion. Right: A simple high-level workflow of Invert2Restore. Our method estimates the initial noise to reconstruct the degraded image. Then, we rectify the initial noise by finding a nearby noise sample with a higher density of the standard normal distribution. Lastly, we apply the deterministic reverse diffusion to obtain the final restored image.
• Figure 3: Images generated by a pre-trained diffusion model (second row) fed with the respective input noise (first row). Left column: the initial noise $z_1$ is drawn from the standard normal. Middle column: $z_2 = \beta_2 z_1$, $\beta_2=0.96<1$. Right column: $z_3 = \beta_3 z_1$, $\beta_3=1.04>1$. Notice that $p_z(z_1)>p_z(z_2)$ and $p_z(z_1)>p_z(z_3)$. Thus, the quality and content of the generated images depend on whether the sampled noise is from a higher ($z_1$) or a low ($z_2$ and $z_3$) density region of $p_z(z) = \mathcal{N}(\mathbf{0}, \mathbf{I})$.
• Figure 4: Qualitative comparisons of different partially blind restoration tasks on CelebA (the degradation operator is known, but not its parameters). First row: motion deblurring. Second row: $4\times$ super-resolution.
• Figure 5: Qualitative comparisons of different restoration tasks where the parametric form of the degradation is unknown. First row: Deraining. Second row: JPEG de-artifacting. Third row: raindrop removal.

Theorems & Definitions (1)

• Proposition 1