InfoBFR: Real-World Blind Face Restoration via Information Bottleneck

TL;DR

InfoBFR addresses real-world blind face restoration by combining information compression and diffusion-based compensation on a manifold. The method uses a Manifold Information Bottleneck to filter neural degradations from pre-trained BFR outputs and a diffusion LoRA-based one-shot denoising to recover high-fidelity facial details, achieving strong generalization across challenging wild datasets. Quantitative and qualitative results demonstrate superiority over state-of-the-art GAN- and diffusion-based BFR methods, with efficient inference and a modest parameter footprint. This plug-and-play approach enables existing BFR models to better resist neural degradations, improving realism and preserving identity in real-world scenarios.

Abstract

Blind face restoration (BFR) is a highly challenging problem due to the uncertainty of data degradation patterns. Current BFR methods have realized certain restored productions but with inherent neural degradations that limit real-world generalization in complicated scenarios. In this paper, we propose a plug-and-play framework InfoBFR to tackle neural degradations, e.g., prior bias, topological distortion, textural distortion, and artifact residues, which achieves high-generalization face restoration in diverse wild and heterogeneous scenes. Specifically, based on the results from pre-trained BFR models, InfoBFR considers information compression using manifold information bottleneck (MIB) and information compensation with efficient diffusion LoRA to conduct information optimization. InfoBFR effectively synthesizes high-fidelity faces without attribute and identity distortions. Comprehensive experimental results demonstrate the superiority of InfoBFR over state-of-the-art GAN-based and diffusion-based BFR methods, with around 70ms consumption, 16M trainable parameters, and nearly 85% BFR-boosting. It is promising that InfoBFR will be the first plug-and-play restorer universally employed by diverse BFR models to conquer neural degradations.

Figures (13)

• Figure 1: InfoBFR helps pre-trained BFR models (CodeFormer codeformer, DiffBIR diffbir, GPEN GPEN) fight against neural degradations, e.g., topology distortion (col 1), prior bias (col 2), texture distortion (col 3), and artifact residues (col 4). Note that pre-trained BFR models and our proposed InfoBFR are both trained on the FFHQ style2019 dataset. More visual examples are shown in Fig. \ref{['fig:stage2']}.
• Figure 1: Visual results of state-of-the-art BRF models and the corresponding InfoBFR results in Wider-Test dataset codeformer.
• Figure 2: Motivation of InfoBFR. In low-quality data distribution, diverse data degradations are caused by physical reasons such as optical imaging of images, external contamination (col 4, Fig \ref{['fig:start']}), or digital processing reasons such as image transmission, image enc-dec. Low-generalization BFR models suffer from severe data degradations and produce high-quality images to a certain extent. However, neural BFR models give birth to new problems, i.e., neural hallucinations during model inference. We define these urgent and neglected degradations as neural degradations. We first study the universal neural degradation restoration method InfoBFR. Specifically, we conduct information compression and compensation where novel details and inherited information from the pre-trained BFR model are neurally integrated.
• Figure 2: Visual results of CodeFormer codeformer, GPEN GPEN, DiffBIR diffbir and corresponding InfoBFR results.
• Figure 3: Comprehensive FID-boosting and MUSIQ-boosting visualization of BFR* models equipped with plug-and-play InfoBFR, compared with the original BFR in real-world datasets with heavy degradations (Wider-Test codeformer, FOS fos, CelebChild gfp2021 and WebPhoto gfp2021). InfoBFR facilitates BFR quality by effectively harnessing the information bottleneck. More quantitative evaluations are shown in Tab. \ref{['tab:improve']} and Tab. \ref{['tab:real']}.