Diff-PC: Identity-preserving and 3D-aware Controllable Diffusion for Zero-shot Portrait Customization

TL;DR

Diff-PC introduces a tuning-free, diffusion-based framework for zero-shot portrait customization that preserves identity while allowing precise control over facial expression, pose, and background. It leverages a 3D face predictor to create a 3D-aware prior, an ID-Encoder to capture fine-grained identity features, and two novel modules—ID-Ctrl and ID-Injector—to align and inject identity information into the diffusion process. A dedicated ID-centric dataset is collected to strengthen ID fidelity and T2I consistency. Empirical results show state-of-the-art performance in identity preservation, facial controllability, and text-to-image alignment, with demonstrated compatibility across multiple SDXL-style foundation models. The work enables rapid, high-fidelity portrait customization suitable for applications like virtual try-on and personalized avatars while highlighting ethical considerations around face synthesis.

Abstract

Portrait customization (PC) has recently garnered significant attention due to its potential applications. However, existing PC methods lack precise identity (ID) preservation and face control. To address these tissues, we propose Diff-PC, a diffusion-based framework for zero-shot PC, which generates realistic portraits with high ID fidelity, specified facial attributes, and diverse backgrounds. Specifically, our approach employs the 3D face predictor to reconstruct the 3D-aware facial priors encompassing the reference ID, target expressions, and poses. To capture fine-grained face details, we design ID-Encoder that fuses local and global facial features. Subsequently, we devise ID-Ctrl using the 3D face to guide the alignment of ID features. We further introduce ID-Injector to enhance ID fidelity and facial controllability. Finally, training on our collected ID-centric dataset improves face similarity and text-to-image (T2I) alignment. Extensive experiments demonstrate that Diff-PC surpasses state-of-the-art methods in ID preservation, facial control, and T2I consistency. Furthermore, our method is compatible with multi-style foundation models.

Figures (21)

• Figure 1: Two examples of portrait customization. Compared to the SOTA method (PhotoMaker-V2), our approach achieves superior identity fidelity, text-to-image consistency, and the ability to specify facial position and expression.
• Figure 2: (a) Previous methods lack facial geometric conditions, obtaining poor results. (b) Our approach leverages the 3D-aware facial prior to preserve identity details and control facial attributes. (c) 3D face predictor synthesizes the 3D face that integrates the reference identity with the target expression and pose.
• Figure 3: The overall training architecture of Diff-PC, which is built upon SDXL (VAE is omitted in the figure). First, we employ the 3D face predictor to reconstruct a 3D facial prior $I_{3d}$ that contains reference ID and target attributes (e.g., expressions and postures). Next, fine-grained ID features $F_{id}^{\prime}$ are obtained via ID-Encoder. Subsequently, we utilize $I_{3d}$ to guide the alignment of $F_{id}^{\prime}$ in ID-Ctrl. Finally, we inject $c_{id}$ into UNet through ID-Injector to accomplish ID preservation and facial control.
• Figure 4: ID-Ctrl and ID-Injector in Diff-PC. For brevity, we only draw two blocks in the encoder and decoder, but they actually have three blocks each.
• Figure 5: The inference pipeline.