Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration

TL;DR

Pref-Restore addresses the ill-posed nature of blind face restoration by pairing an auto-regressive semantic integrator with a diffusion-based generator to densely encode high-level guidance and then sharpening the output distribution through on-policy reinforcement learning. The two-stage training—Stage 1 multi-modal knowledge alignment and Stage 2 DiffusionNFT-based preference optimization—tethers the restoration to a stable facial manifold while suppressing stochastic hallucinations, yielding deterministic, identity-preserving results. The approach achieves state-of-the-art performance on synthetic and real-world benchmarks and significantly reduces solution entropy, enabling reliable and controllable restoration across diverse degradations. This hierarchical, preference-aligned framework provides a practical pathway toward deterministic, high-fidelity face restoration suitable for forensics, media restoration, and other sensitive applications.

Abstract

Blind face restoration remains a persistent challenge due to the inherent ill-posedness of reconstructing holistic structures from severely constrained observations. Current generative approaches, while capable of synthesizing realistic textures, often suffer from information asymmetry -- the intrinsic disparity between the information-sparse low quality inputs and the information-dense high quality outputs. This imbalance leads to a one-to-many mapping, where insufficient constraints result in stochastic uncertainty and hallucinatory artifacts. To bridge this gap, we present \textbf{Pref-Restore}, a hierarchical framework that integrates discrete semantic logic with continuous texture generation to achieve deterministic, preference-aligned restoration. Our methodology fundamentally addresses this information disparity through two complementary strategies: (1) Augmenting Input Density: We employ an auto-regressive integrator to reformulate textual instructions into dense latent queries, injecting high-level semantic stability to constrain the degraded signals; (2) Pruning Output Distribution: We pioneer the integration of on-policy reinforcement learning directly into the diffusion restoration loop. By transforming human preferences into differentiable constraints, we explicitly penalize stochastic deviations, thereby sharpening the posterior distribution toward the desired high-fidelity outcomes. Extensive experiments demonstrate that Pref-Restore achieves state-of-the-art performance across synthetic and real-world benchmarks. Furthermore, empirical analysis confirms that our preference-aligned strategy significantly reduces solution entropy, establishing a robust pathway toward reliable and deterministic blind restoration.

Figures (14)

• Figure 1: Conceptual illustration of (a) the conventional generative prior paradigm versus (b) our proposed Pref-Restore framework. Existing methods suffer from information asymmetry, where ill-posed priors and sparse inputs lead to stochastic outcomes such as hallucinations or identity loss. Our Pref-Restore resolves this by augmenting input density through AR-based semantic modeling and pruning the output distribution via on-policy reinforcement learning, effectively pruning the uncertain solution space to achieve deterministic, preference-consistent restoration.
• Figure 2: The overall framework of Pref-Restore.(a) Hierarchical Restoration Architecture: Our model decouples the blind face restoration task into a discrete semantic stream and a continuous texture stream. The AR-based Semantic Integrator processes degraded observations $y$ and textual instructions $T$ to generate discrete semantic tokens $\mathbf{S}$ via Next-Token Prediction (NTP), acting as a global structural anchor. Simultaneously, the Continuous Diffusion-based Generator leverages these semantic anchors and low-level texture latents $\mathbf{z}_{low}$ to reconstruct high-fidelity details through a conditional flow matching process. (b) Preference-Aware Fine-tuning via DiffusionNFT: To eliminate hallucinations, we employ an on-policy RL strategy. The current policy $\mathbf{v}_\theta$ performs a group rollout to generate $K$ candidates, evaluated by a frozen reward model $\mathcal{R}_{pref}$. Based on the normalized rewards $r$, we construct implicit positive ($\mathbf{v}_\theta^+$) and negative ($\mathbf{v}_\theta^-$) velocity proxies. The model is optimized by contrasting these proxies against the forward data flow, rotating the vector field toward the preference-aligned manifold.
• Figure 3: Evolution of FID scores during the hierarchical training process. The curves illustrate the FID (HQ) and FID (FFHQ) results on the CelebA-Test dataset. Stage 1 comprises two critical steps: Step 1 (Semantic-to-Diffusion Alignment) and Step 2 (Texture-to-Diffusion Alignment). The sustained decrease in FID during Stage 1 highlights the scaling effect of our AR-based semantic integrator in aligning distributions. Our framework consistently maintains a downward trajectory, eventually outperforming strong baselines like DifFace and RestoreFormer++ by a clear margin.
• Figure 4: Qualitative comparison of face restoration on the CelebA-HQ dataset. We compare our Pref-Restore (F) and (Q) variants with state-of-the-art methods. Our models demonstrate superior performance in recovering structural details and high-frequency hair textures compared to baselines like CodeFormer and VQFR, which often suffer from over-smoothing in complex regions. The (Q) variant further enhances aesthetic clarity and texture realism through preference-aware RL optimization.
• Figure 5: Visual comparison illustrating the impact of multi-modal text guidance. We present three cases where the degradation is severe. Without text guidance (w.o. text), the model relies solely on the ambiguous visual priors, leading to critical semantic failures: (Top) Misinterpreting "curly blonde hair" as straight hair; (Middle) Hallucinating a female face for a male subject (Identity Error); (Bottom) Losing the "smiling with teeth visible" attribute.