SPIRE: Semantic Prompt-Driven Image Restoration

TL;DR

SPIRE tackles fine-grained, language-guided image restoration by introducing a unified framework that conditions on degraded input $y$, semantic prompt $c_s$, and restoration prompt $c_r$ to produce high-fidelity restorations. It decouples the semantic prior from restoration cues, freezing $p(z_t|c_s)$ while learning $p(y|z_t,c_r)$ through a parameter-efficient ControlNet adaptor that fuses restoration features into a frozen latent diffusion backbone via a modulation layer. The approach leverages a parameterized degradation pipeline and CLIP-based restoration encoding to enable precise control over restoration strength and type, including unseen degradations, and demonstrates superiority over baselines on synthetic and real-world benchmarks. Overall, SPIRE delivers a practical, instruction-based restoration paradigm with flexible test-time prompting and a new benchmark for low-level multimodal image restoration.

Abstract

Text-driven diffusion models have become increasingly popular for various image editing tasks, including inpainting, stylization, and object replacement. However, it still remains an open research problem to adopt this language-vision paradigm for more fine-level image processing tasks, such as denoising, super-resolution, deblurring, and compression artifact removal. In this paper, we develop SPIRE, a Semantic and restoration Prompt-driven Image Restoration framework that leverages natural language as a user-friendly interface to control the image restoration process. We consider the capacity of prompt information in two dimensions. First, we use content-related prompts to enhance the semantic alignment, effectively alleviating identity ambiguity in the restoration outcomes. Second, our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength, without the need for explicit task-specific design. In addition, we introduce a novel fusion mechanism that augments the existing ControlNet architecture by learning to rescale the generative prior, thereby achieving better restoration fidelity. Our extensive experiments demonstrate the superior restoration performance of SPIRE compared to the state of the arts, alongside offering the flexibility of text-based control over the restoration effects.

Figures (15)

• Figure 1: We present SPIRE: Semantic Prompt-Driven Image Restoration , a text-based foundation model for all-in-one, instructed image restoration. SPIRE allows users to flexibly leverage either semantic-level content prompt, or quantitative degradation-aware restoration prompt, or both, to obtain their desired enhancement results based on personal preferences. In other words, SPIRE can be easily prompted to conduct blind restoration, semantic restoration, or task-specific granular treatment. Our framework also enables a new paradigm of instruction-based image restoration, providing a reliable evaluation benchmark to facilitate vision-language models for low-level computational photography applications.
• Figure 2: Framework of SPIRE. In the training phase, we begin by synthesizing a degraded version $y$, of a clean image $x$. Our degradation synthesis pipeline also creates a restoration prompt $\bm{c}_r$ , which contains numeric parameters that reflects the intensity of the degradation introduced. Then, we inject the synthetic restoration prompt into a ControlNet adaptor, which uses our proposed modulation fusion blocks ($\gamma$, $\beta$) to connect with the frozen backbone driven by the semantic prompt $\bm{c}_s$. During test time, the users can employ the SPIRE framework as either a blind restoration model with restoration prompt "Remove all degradation" and empty semantic prompt $\varnothing$, or manually adjust the restoration $\bm{c}_r$ and semantic prompts $\bm{c}_s$ to obtain what they want.
• Figure 3: Degradation ambiguities in real-world problems. By adjusting the restoration prompt, our method can preserve the motion effect that is coupled with Gaussian blur, while fully blind restoration models do not provide this flexibility.
• Figure 4: Visual Comparison with other baselines. Our method of integrating both the semantic prompt $\boldsymbol{c}_s{}$ and the restoration prompt $\boldsymbol{c}_r{}$ outperforms imge-to-image restoration (DiffBIR, Retrained ControlNet-SR) and naive zero-shot combination with semantic prompt. It achieves more sharp and clean results while maintaining consistency with the degraded image.
• Figure 5: Real-world image restorations. Our framework enhances real-world low-quality images following semantic prompts provided by language models or user input.