Collaborative Face Experts Fusion in Video Generation: Boosting Identity Consistency Across Large Face Poses

TL;DR

The paper tackles identity drift in video generation under large face poses by introducing Collaborative Face Experts Fusion (CoFE), a layer-aware fusion of pose-robust identity, semantic, and detail features into Diffusion Transformer backbones. It pairs this with LaFID-180K, a pose-annotated, high-quality dataset produced via a Face Constraints, Identity Consistency, and Speech Disambiguation pipeline, and LaFID-Bench for rigorous large-pose evaluation. Results on LaFID-Bench and VIP-Test show state-of-the-art improvements in face similarity, CLIP semantic alignment, and overall video quality, highlighting the importance of multi-source facial priors and pose-aware data for robust identity preservation. The work provides strong evidence that dynamic, task-aligned fusion of disentangled facial representations significantly enhances identity stability across challenging poses in video synthesis.

Abstract

Current video generation models struggle with identity preservation under large face poses, primarily facing two challenges: the difficulty in exploring an effective mechanism to integrate identity features into DiT architectures, and the lack of targeted coverage of large face poses in existing open-source video datasets. To address these, we present two key innovations. First, we propose Collaborative Face Experts Fusion (CoFE), which dynamically fuses complementary signals from three specialized experts within the DiT backbone: an identity expert that captures cross-pose invariant features, a semantic expert that encodes high-level visual context, and a detail expert that preserves pixel-level attributes such as skin texture and color gradients. Second, we introduce a data curation pipeline comprising three key components: Face Constraints to ensure diverse large-pose coverage, Identity Consistency to maintain stable identity across frames, and Speech Disambiguation to align textual captions with actual speaking behavior. This pipeline yields LaFID-180K, a large-scale dataset of pose-annotated video clips designed for identity-preserving video generation. Experimental results on several benchmarks demonstrate that our approach significantly outperforms state-of-the-art methods in face similarity, FID, and CLIP semantic alignment. Project page: https://rain152.github.io/CoFE/.

Figures (8)

• Figure 1: Top: Identity preservation under large face poses is challenging in current methods, while our approach shows clear gains in face similarity metrics. Bottom: With dynamic multi-feature injection, our proposed method achieves the best or competitive performance across all metrics.
• Figure 2: Overview of our pipeline. Our CoFE module comprises globally shared projection layers to align different face features into the feature space of DiT and per-DiT-block expert fusion layers for dynamic feature fusion. Specifically, we employ ArcFace, CLIP, and DINOv2 as our identity, semantic, and detail encoders, respectively.
• Figure 3: Cross-attention visualizations of different experts. Warmer colors denote higher attention strength. CoFE integrates complementary attention from Identity Expert (ArcFace), Semantic Expert (CLIP), and Detail Expert (DINOv2) to jointly enhance facial detail fidelity and identity consistency.
• Figure 4: The statistical analysis of the LaFID-180K dataset.
• Figure 5: Comparison of video quality metrics between LaFID-180K and OpenHumanVid. LaFID-180K exhibits better quality.