MagicPortrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance

TL;DR

This work tackles video face reenactment by addressing shape preservation and temporal coherence. It introduces MagicPortrait, which uses the FLAME 3D parametric face model to derive motion guidance and couples depth, normal, and rendering maps with a Geometric Guidance Encoder to condition a latent diffusion model. The approach achieves strong identity fidelity, accurate expression and head-pose transfer, and robust generalization on out-of-domain data, validated on CelebV-HQ and cross-domain images. The combination of Structured Face Alignment, multi-level feature fusion with self-attention, and two-stage training yields temporally coherent, high-quality face animations with practical impact for digital content creation.

Abstract

In this study, we propose a method for video face reenactment that integrates a 3D face parametric model into a latent diffusion framework, aiming to improve shape consistency and motion control in existing video-based face generation approaches. Our approach employs the FLAME (Faces Learned with an Articulated Model and Expressions) model as the 3D face parametric representation, providing a unified framework for modeling face expressions and head pose. This not only enables precise extraction of motion features from driving videos, but also contributes to the faithful preservation of face shape and geometry. Specifically, we enhance the latent diffusion model with rich 3D expression and detailed pose information by incorporating depth maps, normal maps, and rendering maps derived from FLAME sequences. These maps serve as motion guidance and are encoded into the denoising UNet through a specifically designed Geometric Guidance Encoder (GGE). A multi-layer feature fusion module with integrated self-attention mechanisms is used to combine facial appearance and motion latent features within the spatial domain. By utilizing the 3D face parametric model as motion guidance, our method enables parametric alignment of face identity between the reference image and the motion captured from the driving video. Experimental results on benchmark datasets show that our method excels at generating high-quality face animations with precise expression and head pose variation modeling. In addition, it demonstrates strong generalization performance on out-of-domain images. Code is publicly available at https://github.com/weimengting/MagicPortrait.

Figures (5)

• Figure 1: Given a sequence of motion signals, MagicPortrait (ours) is capable of generating temporally coherent animations for reference face images. In contrast, state-of-the-art approaches struggle to maintain appearance details or transfer precise facial expressions. The motion sequence is shown in the corner for reference. *Note: HyperReenact utilizes raw video frames directly as the driving signal. Both Champ and our method utilize conditions extracted from the same driving video as used in HyperReenact.
• Figure 2: Qualitative comparisons on self-reenactment on samples from CelebV-HQ dataset.
• Figure 3: Ablation analysis on different motion conditions. Lmks, dep. and nor. stand for landmarks, depth and normal, respectively. Zoom in to compare the details.
• Figure 4: Ablation analysis on different architecture of Geometric Guidance Encoder. w/o. indicate the guidance without self-attention.
• Figure 5: The motion conditions alongside their corresponding self-attention maps. The left column of each group showcases the depth map, normal map, and rendering map generated from the associated FLAME sequences. The right column displays the resulting self-attention outputs obtained from the guidance encoder.