DisControlFace: Adding Disentangled Control to Diffusion Autoencoder for One-shot Explicit Facial Image Editing

TL;DR

DisControlFace tackles the challenge of explicit, fine-grained facial editing by disentangling high-level semantic conditioning from explicit 3DMM-based controls within a diffusion framework. It freezes a Diffusion Autoencoder to preserve deterministic semantic conditioning and introduces Exp-FaceNet to provide spatially aligned $3DMM$-driven control, trained via a Random Semantic Masking strategy to enable independent editing. A one-shot fine-tuning step injects subject-specific priors, improving identity preservation while editing explicit attributes such as pose, expression, and lighting, and enabling inpainting and semantic manipulations. The approach achieves state-of-the-art results in one-shot facial editing on in-the-wild images, while highlighting limitations related to data scale, geometric fidelity of eyeballs, and inference speed, suggesting avenues for future improvements in speed and 3D accuracy.

Abstract

In this work, we focus on exploring explicit fine-grained control of generative facial image editing, all while generating faithful facial appearances and consistent semantic details, which however, is quite challenging and has not been extensively explored, especially under an one-shot scenario. We identify the key challenge as the exploration of disentangled conditional control between high-level semantics and explicit parameters (e.g., 3DMM) in the generation process, and accordingly propose a novel diffusion-based editing framework, named DisControlFace. Specifically, we leverage a Diffusion Autoencoder (Diff-AE) as the semantic reconstruction backbone. To enable explicit face editing, we construct an Exp-FaceNet that is compatible with Diff-AE to generate spatial-wise explicit control conditions based on estimated 3DMM parameters. Different from current diffusion-based editing methods that train the whole conditional generative model from scratch, we freeze the pre-trained weights of the Diff-AE to maintain its semantically deterministic conditioning capability and accordingly propose a random semantic masking (RSM) strategy to effectively achieve an independent training of Exp-FaceNet. This setting endows the model with disentangled face control meanwhile reducing semantic information shift in editing. Our model can be trained using 2D in-the-wild portrait images without requiring 3D or video data and perform robust editing on any new facial image through a simple one-shot fine-tuning. Comprehensive experiments demonstrate that DisControlFace can generate realistic facial images with better editing accuracy and identity preservation over state-of-the-art methods. Project page: https://discontrolface.github.io/

Figures (12)

• Figure 1: The reconstruction and editing results of using different initial noise map computing strategy in Diff-AE. $X_T$ indicates using the original reverse deterministic computing to generate the initial noise map of Diff-AE.
• Figure 2: Pipeline overview. Our DisControlFace leverages Diffusion Autoencoder (Diff-AE) as the reconstruction backbone freeze its pre-trained weights to maintain the semantic deterministic conditioning capability, which is effective in reducing semantic information shift during the editing of the input portrait image. Then, an explicit face control network, Exp-FaceNet compatible with the Diff-AE is constructed, which takes pixel-aligned snapshots rendered from estimated explicit parameters as inputs and generates multi-scale control features to condition the DDIM decoder. Moreover, a random semantic masking (RSM) training strategy is accordingly designed to enable a disentangled explicit face control of Exp-FaceNet.
• Figure 2: Ablation study on fine-grained face parameters. Adopting the detail vector $\theta$ estimated by EMOCA danvevcek2022emoca can help to generate the control condition with more fine-grained face geometry, which allows faithful facial details preservation.
• Figure 3: Qualitative comparison against baselines in one-shot editing. For each selected image, we use EMOCA danvevcek2022emoca to estimate the corresponding explicit parameters, then synthesize the edited images using different methods based on the modified parameters of pose, expression, and lighting. We additionally provide the rendered shading shapes in the second row as the references of explicit control conditions. As can be seen, our DisControlFace can edit images that match well with the target control conditions while faithfully synthesizing facial appearances and editing-irrelevant details.
• Figure 3: Ablation study on different masking strategies in inference. We set the inference denoising steps to 20 for all masking strategies. Strategy A: the masing ratio is set to 0% for all 20 steps; Strategy B: the masking ratio is set to 75% for all 20 steps; Strategy C: the masking ratio is set to 25% for all 20 steps; Strategy D: the masking ratio is set to 25% and 75% for the first 10 steps and last 10 steps; Strategy E: the masking ratio is set to 75% and 25% for the first 10 steps and last 10 steps: Strategy F: the linear masking ratio introduced in the main paper.