SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration

TL;DR

SSP-IR addresses the challenge of realistic image restoration under real-world degradations by leveraging semantic and structure priors within a diffusion-based framework. It combines explicit textual guidance from a multimodal large language model with implicit image semantics via CLIP embeddings, and enforces degradation-independent structure control through a constrained Processor and ControlNet, all integrated by a Prior-Guided Attention module. The approach is trained in one stage with end-to-end optimization and losses that preserve both local details and global structure, achieving state-of-the-art performance on synthetic and real datasets in semantic fidelity and perceptual quality. The work demonstrates the potential of integrating MLLMs and multi-level priors into diffusion-based restoration, while noting higher computational requirements and latency as practical considerations for deployment.

Abstract

Realistic image restoration is a crucial task in computer vision, and diffusion-based models for image restoration have garnered significant attention due to their ability to produce realistic results. Restoration can be seen as a controllable generation conditioning on priors. However, due to the severity of image degradation, existing diffusion-based restoration methods cannot fully exploit priors from low-quality images and still have many challenges in perceptual quality, semantic fidelity, and structure accuracy. Based on the challenges, we introduce a novel image restoration method, SSP-IR. Our approach aims to fully exploit semantic and structure priors from low-quality images to guide the diffusion model in generating semantically faithful and structurally accurate natural restoration results. Specifically, we integrate the visual comprehension capabilities of Multimodal Large Language Models (explicit) and the visual representations of the original image (implicit) to acquire accurate semantic prior. To extract degradation-independent structure prior, we introduce a Processor with RGB and FFT constraints to extract structure prior from the low-quality images, guiding the diffusion model and preventing the generation of unreasonable artifacts. Lastly, we employ a multi-level attention mechanism to integrate the acquired semantic and structure priors. The qualitative and quantitative results demonstrate that our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets. Our project page is https://zyhrainbow.github.io/projects/SSP-IR.

Figures (11)

• Figure 1: Illustration of our method's superiority compared with other diffusion-based methods. (a) shows our method's good performance on the semantic, structure and perception quality. We calculate the CLIP similarity between the generated results and GT images to evaluate the semantic fidelity, SSIM to evaluate the structure accuracy and CLIPIQA to evaluate the perception quality. All metrics are calculated on RealSR dataset cai2019realsr and are the higher the better. Our method achieves a good balance about semantic-structure-perception. In the figure, $\checkmark$ indicates better performance and $\times$ indicates poorer performance. (b) shows the visual comparison compared with other diffusion-based methods.
• Figure 2: The framework of the proposed method, which consists of semantic prior extraction module, structure prior extraction module and the denoising U-Net. Semantic prior extraction module consists of two branches and is used to generate explicit text embedding and implicit image embedding. Structure prior extraction module is used to generate degradation-independent structure control. The denoising U-Net integrates the text embedding, image embedding and strcuture control into the denoising process through the Prior-Guided Attention module.
• Figure 3: The comparison of different prompts and their corresponding restoration results with PASD yang2023pixel. (a) -(d) show the null prompt, BLIP prompt from LQ image, BLIP prompt from GT image and MLLM prompt predicted from LQ image and its corresponding restoration result. MLLM prompt is "A field of ripe wheat stands against a clear blue sky, creating a picturesque sight. The golden wheat glows under the sunlight, while the serene blue sky provides a beautiful backdrop."
• Figure 4: Qualitative comparisons with different state-of-the-art methods on synthetic datasets. We have provided the magnified regions for clarity. Please zoom in for a better view.
• Figure 5: Qualitative comparisons with different state-of-the-art methods on real-world datasets. We have provided the magnified regions for clarity. Please zoom in for a better view.