DR-BFR: Degradation Representation with Diffusion Models for Blind Face Restoration

TL;DR

This paper equips diffusion models with the capability to decouple various degradation as a degradation prompt from low-quality (LQ) face images via unsupervised contrastive learning with reconstruction loss, and demonstrates that this capability significantly improves performance, particularly in terms of the naturalness of the restored images.

Abstract

Blind face restoration (BFR) is fundamentally challenged by the extensive range of degradation types and degrees that impact model generalization. Recent advancements in diffusion models have made considerable progress in this field. Nevertheless, a critical limitation is their lack of awareness of specific degradation, leading to potential issues such as unnatural details and inaccurate textures. In this paper, we equip diffusion models with the capability to decouple various degradation as a degradation prompt from low-quality (LQ) face images via unsupervised contrastive learning with reconstruction loss, and demonstrate that this capability significantly improves performance, particularly in terms of the naturalness of the restored images. Our novel restoration scheme, named DR-BFR, guides the denoising of Latent Diffusion Models (LDM) by incorporating Degradation Representation (DR) and content features from LQ images. DR-BFR comprises two modules: 1) Degradation Representation Module (DRM): This module extracts degradation representation with content-irrelevant features from LQ faces and estimates a reasonable distribution in the degradation space through contrastive learning and a specially designed LQ reconstruction. 2) Latent Diffusion Restoration Module (LDRM): This module perceives both degradation features and content features in the latent space, enabling the restoration of high-quality images from LQ inputs. Our experiments demonstrate that the proposed DR-BFR significantly outperforms state-of-the-art methods quantitatively and qualitatively across various datasets. The DR effectively distinguishes between various degradations in blind face inverse problems and provides a reasonably powerful prompt to LDM.

Figures (9)

• Figure 1: Comparison of guidance mechanisms between previous diffusion-based methods and our DR-BFR. The key distinction lies in the approach used to handle LQ face images. In our DR-BFR, the DR features are decoupled in advance. These decoupled features as a resonably prompt, combined with the extracted content features, serve as guiding conditions for DM.
• Figure 2: Framework of the proposed DR-BFR for blind face restoration task. DR-BFR is fundamentally a multi-input conditional diffusion model with specific guidance. Given the LQ face, DRM decouples the content-irrelevant degradation information, while an encoder, inherited from LDRM, extracts the content of the LQ image. The diffusion-based design of DR-BFR, which incorporates degradation awareness, demonstrates superior performance as validated by experimental evidence.
• Figure 3: Training graph of the Degradation Representation Module. Here, $\hat{y}_{LQ}$ represents the LQ image generated by the reconstruction of DR and HQ image. Red brackets indicate positive samples within the same batch, while blue brackets denote negative samples.
• Figure 4: Qualitative comparisons on the CelebA-Test for blind face restoration. Our DR-BFR demonstrates strong performance in detail enhancement, hue preservation and attitude preservation compared to these latest GAN-based, dictionary-based, and diffusion-based methods. Restore denotes RestoreFormer. Zoom in for best view.
• Figure 5: Qualitative comparisons on real-world datasets. Our DR-BFR demonstrates superior performance in both detail enhancement and hue preservation, particularly on inputs with severe degradation. Zoom in for best view.