FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation

TL;DR

The paper addresses identity-preserving text-to-video generation (IPT2V) by enhancing facial knowledge in pre-trained diffusion transformers. It introduces FantasyID, a tuning-free framework that couples a 3D facial prior from DECA with multi-view augmentation and a layer-aware, adaptive feature injection to guide DiT-based video synthesis. A fusion transformer integrates 2D appearance tokens with 3D vertex features to produce an identity descriptor, which is injected into the model at per-layer granularity. Experiments across diverse datasets show improved identity preservation, facial dynamics, and visual quality over existing tuning-free IPT2V methods, with ablations validating each component’s contribution.

Abstract

Tuning-free approaches adapting large-scale pre-trained video diffusion models for identity-preserving text-to-video generation (IPT2V) have gained popularity recently due to their efficacy and scalability. However, significant challenges remain to achieve satisfied facial dynamics while keeping the identity unchanged. In this work, we present a novel tuning-free IPT2V framework by enhancing face knowledge of the pre-trained video model built on diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry prior is incorporated to ensure plausible facial structures during video synthesis. To prevent the model from learning copy-paste shortcuts that simply replicate reference face across frames, a multi-view face augmentation strategy is devised to capture diverse 2D facial appearance features, hence increasing the dynamics over the facial expressions and head poses. Additionally, after blending the 2D and 3D features as guidance, instead of naively employing cross-attention to inject guidance cues into DiT layers, a learnable layer-aware adaptive mechanism is employed to selectively inject the fused features into each individual DiT layers, facilitating balanced modeling of identity preservation and motion dynamics. Experimental results validate our model's superiority over the current tuning-free IPT2V methods.

Figures (6)

• Figure 1: Examples of FantasyID. Given a human face image, FantasyID generates ID-preserving videos with enhanced motion dynamics and more stable facial structures.
• Figure 2: Overview of FantasyID. The framework constructs a multi-view face collection, randomly selects one face as the reference input, and employs face abstractor to extract 2D visual tokens while using DECA to extract 3D face vertex tokens. We fuse both the 2D and 3D tokens with fusion transformer layers and guide DiT-based model via a layer-aware signal injection method.
• Figure 3: Qualitiative Comparision between our methods and ConsisID, ID-Animator. Please refer our supplementary materials for the video results.
• Figure 4: Effect of 3D Constraints. By altering the facial widths and jawline sharpness of face vertex, the generated facial videos exhibit noticeable structural changes.
• Figure 5: Ablation study on Face Vertex(FV). Without the face vertex leading to distorted facial structures during motion.