Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration

TL;DR

This work tackles universal image restoration under diverse real-world degradations by leveraging pre-trained Stable Diffusion priors and CLIP-based visual prompts. It introduces Diff-Restorer, which extracts semantic and degradation prompts via a Visual Prompt Processor to guide a diffusion backbone, and employs an Image-guided Control Module plus a Degradation-aware Decoder to preserve spatial structure and refine details in the latent-to-pixel mapping. The method demonstrates strong perceptual restoration performance across eight degradation types, real-world scenarios, and mixed degradations, with comprehensive ablations highlighting the contributions of semantic guidance, degradation-aware control, and decoder refinement. By exploiting diffusion priors and adaptive visual prompts, the approach advances toward a practical, general-purpose restoration system with high realism and fidelity.

Abstract

Image restoration is a classic low-level problem aimed at recovering high-quality images from low-quality images with various degradations such as blur, noise, rain, haze, etc. However, due to the inherent complexity and non-uniqueness of degradation in real-world images, it is challenging for a model trained for single tasks to handle real-world restoration problems effectively. Moreover, existing methods often suffer from over-smoothing and lack of realism in the restored results. To address these issues, we propose Diff-Restorer, a universal image restoration method based on the diffusion model, aiming to leverage the prior knowledge of Stable Diffusion to remove degradation while generating high perceptual quality restoration results. Specifically, we utilize the pre-trained visual language model to extract visual prompts from degraded images, including semantic and degradation embeddings. The semantic embeddings serve as content prompts to guide the diffusion model for generation. In contrast, the degradation embeddings modulate the Image-guided Control Module to generate spatial priors for controlling the spatial structure of the diffusion process, ensuring faithfulness to the original image. Additionally, we design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain. We conducted comprehensive qualitative and quantitative analysis on restoration tasks with different degradations, demonstrating the effectiveness and superiority of our approach.

Figures (13)

• Figure 1: Examples of the generated images with SD img2img. The left image of each subfigure is generated by the full prompt. The right image is translated from the left image under the guidance of the prompt removing the degradation-related description highlighted in red. (a) is an example of deblurring correlation, while (b) is an example of desnowing. SD img2img can achieve degradation removal but result in changes to the image content.
• Figure 2: Illustration of the similarity between degraded images and degradation-related texts. (a)-(d) represent four different degradation types, and their similarity with four texts ("blurry photo", "low-light photo", "hazy photo", and "snowy photo") is shown in the middle.
• Figure 3: Overview of the architecture of Diff-Restorer. CLIP Image Encoder and Prompt Processor are used to extract visual prompts, which include semantic embedding and degradation embedding. A Image-guided Control Module modulated with degradation embedding is used to provide control information. The pre-trained denoising U-Net utilizes control information and semantic embedding as conditions to denoise. After multiple denoising steps, the latent features are generated and subsequently transformed into high-quality restored images by the Degradation-aware Decoder.
• Figure 4: The architecture of the proposed Degradation Modulation Block (DMB) in the Control Encoder.
• Figure 5: The training strategy of our Degradation-aware Decoder and the architecture of the proposed Degradation-aware Refinement Block (DRB).