FaithDiff: Unleashing Diffusion Priors for Faithful Image Super-resolution

TL;DR

FaithDiff addresses faithful image super-resolution by unleashing diffusion priors from latent diffusion models and aligning degraded inputs with the diffusion process. It introduces an LQ feature extractor based on a VAE encoder, an alignment module to fuse degraded latent information with diffusion latents, and a CLIP-guided, text-conditioned diffusion framework trained in a unified end-to-end manner. The method achieves state-of-the-art performance on synthetic and real-world benchmarks, delivering higher structural fidelity and better OCR reliability than prior GAN- and diffusion-based SR methods. This work demonstrates that jointly optimizing the encoder and diffusion model, with alignment and text guidance, can robustly separate degradation effects from true image content and produce faithful, high-quality SR outputs.

Abstract

Faithful image super-resolution (SR) not only needs to recover images that appear realistic, similar to image generation tasks, but also requires that the restored images maintain fidelity and structural consistency with the input. To this end, we propose a simple and effective method, named FaithDiff, to fully harness the impressive power of latent diffusion models (LDMs) for faithful image SR. In contrast to existing diffusion-based SR methods that freeze the diffusion model pre-trained on high-quality images, we propose to unleash the diffusion prior to identify useful information and recover faithful structures. As there exists a significant gap between the features of degraded inputs and the noisy latent from the diffusion model, we then develop an effective alignment module to explore useful features from degraded inputs to align well with the diffusion process. Considering the indispensable roles and interplay of the encoder and diffusion model in LDMs, we jointly fine-tune them in a unified optimization framework, facilitating the encoder to extract useful features that coincide with diffusion process. Extensive experimental results demonstrate that FaithDiff outperforms state-of-the-art methods, providing high-quality and faithful SR results.

Figures (6)

• Figure 1: Visual comparison with state-of-the-art SR methods. (b) and (c) are the intermediate results of the degradation removal module (DRM) by DiffBIR DiffBIR and SUPIR SUPIR. The competing methods DiffBIRSUPIR do not effectively restore faithful structures based on the degradation removal results in (b)-(c). In contrast to the results in (d) and (f)-(g), our approach can recover more realistic high-quality results with faithful contents (e.g., the characters in (h)).
• Figure 2: An overview of the FaithDiff, which takes LQ images and image descriptions as inputs and restores HQ images via diffusion process. To fully leverage the power of LDMs, we propose to unleash diffusion priors. An alignment module is developed to effectively incorporate the features extracted from LQ images with the noisy latent of the diffusion model. We jointly optimize the encoder, the alignment module, and the diffusion model, which can benefit from their interplay and lead to faithful SR images with high viusal quality.
• Figure 3: Image SR result (×4) on the synthetic benchmark. The restored image by GAN-based methods Real-ESRGAN exhibits perceptually unpleasant artifacts in (b). Existing diffusion-based methods StableSRDiffBIRSeeSRSUPIR over-smooth the details in (c) or generate incorrect structures in (d) and (f)-(g). In contrast, the proposed method recovers much clearer images with faithful structures in (h).
• Figure 4: Image SR result (×2) on the real-world benchmarks. Compared to competing methods, our approach generates more realistic images with fine-scale structures and details.
• Figure 5: Effectiveness of the unified feature optimization on image SR ($\times 4$). Using unify optimization strategy is able to generate the results with clearer structural details.