XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution

TL;DR

This work tackles diffusion-based image super-resolution under real-world degradations by leveraging cross-modal priors from Multimodal Large Language Models to guide generation. It introduces XPSR, which uses high- and low-level semantic prompts from MLLMs and fuses them with the diffusion prior via a Semantic-Fusion Attention module, while enforcing semantic preservation through a Degradation-Free Constraint. Training freezes the base diffusion model and optimizes ControlNet and Conditional Attention with a diffusion loss $L_{D}$ and a degradation-consistency loss $L_{DFC}$, employing classifier-free guidance during inference with negative prompts. Across synthetic and real-world datasets, XPSR achieves state-of-the-art perceptual quality and semantic fidelity, outperforming prior GAN- and diffusion-based ISR methods on non-reference IQA metrics such as MANIQA, MUSIQ, and CLIPIQA, with qualitatively richer and more semantically accurate restorations.

Abstract

Diffusion-based methods, endowed with a formidable generative prior, have received increasing attention in Image Super-Resolution (ISR) recently. However, as low-resolution (LR) images often undergo severe degradation, it is challenging for ISR models to perceive the semantic and degradation information, resulting in restoration images with incorrect content or unrealistic artifacts. To address these issues, we propose a \textit{Cross-modal Priors for Super-Resolution (XPSR)} framework. Within XPSR, to acquire precise and comprehensive semantic conditions for the diffusion model, cutting-edge Multimodal Large Language Models (MLLMs) are utilized. To facilitate better fusion of cross-modal priors, a \textit{Semantic-Fusion Attention} is raised. To distill semantic-preserved information instead of undesired degradations, a \textit{Degradation-Free Constraint} is attached between LR and its high-resolution (HR) counterpart. Quantitative and qualitative results show that XPSR is capable of generating high-fidelity and high-realism images across synthetic and real-world datasets. Codes are released at \url{https://github.com/qyp2000/XPSR}.

Figures (12)

• Figure 1: Impact of textual prompts for conditional T2I diffusion models. (a) When given high-level prompts containing object categories and detailed textures, high-realism HR images can be restored based on ambiguous LR images. (b) When furnished with accurate low-level prompts that encompass distortion types or the general quality of LR images, high-fidelity images can be generated from blurry or noisy inputs.
• Figure 1: Limitations of diffusion-based methods. Due to their limited semantic understanding, the restored content may be unrelated to the original image.
• Figure 2: Framework of XPSR. (a) First, we integrate an MLLM to acquire semantic priors, encompassing both high-level and low-level descriptions for the LR image. Two varieties of embeddings are derived upon input into the CLIP text encoder. (b) Next, the LR image, along with the embeddings above, are input into the controlled diffusion model as conditions through a Semantic-Fusion Attention (SFA), adhering to the defined workflow. Besides, a Degradation-Free Constraint (DFC) is appended to the ControlNet part, alleviating the challenge of discerning distortions.
• Figure 2: Qualitative comparisons with different SOTA methods on real-world images. Zoom in for a better view.
• Figure 3: Given appropriate instructions, LLaVA can generate high- and low-level semantic prompts consistent with human perception for both high- and low-quality images.