Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model

TL;DR

The paper tackles the challenge of maintaining identity and natural motion in long-form TalkingFace generation. It introduces the Motion-priors Conditional Diffusion Model (MCDM), which fuses archived-clip priors, present-clip diffusion-based motion prediction, and a memory-efficient temporal attention mechanism to sustain temporal coherence. A new TalkingFace-Wild dataset (over 200 hours across 10 languages) supports evaluation and development. Experimental results show state-of-the-art performance in identity preservation, lip synchronization, and motion continuity, with comprehensive ablations validating each component. The work provides code, models, and datasets to advance long-term TalkingFace research while acknowledging ethical considerations for deployment.

Abstract

Recent advances in conditional diffusion models have shown promise for generating realistic TalkingFace videos, yet challenges persist in achieving consistent head movement, synchronized facial expressions, and accurate lip synchronization over extended generations. To address these, we introduce the \textbf{M}otion-priors \textbf{C}onditional \textbf{D}iffusion \textbf{M}odel (\textbf{MCDM}), which utilizes both archived and current clip motion priors to enhance motion prediction and ensure temporal consistency. The model consists of three key elements: (1) an archived-clip motion-prior that incorporates historical frames and a reference frame to preserve identity and context; (2) a present-clip motion-prior diffusion model that captures multimodal causality for accurate predictions of head movements, lip sync, and expressions; and (3) a memory-efficient temporal attention mechanism that mitigates error accumulation by dynamically storing and updating motion features. We also release the \textbf{TalkingFace-Wild} dataset, a multilingual collection of over 200 hours of footage across 10 languages. Experimental results demonstrate the effectiveness of MCDM in maintaining identity and motion continuity for long-term TalkingFace generation. Code, models, and datasets will be publicly available.

Figures (7)

• Figure 1: Our MCDM architecture. On the upper, the archived-clip motion-prior leverages frame-aligned attention with archived-clip, enhancing identity coherence over extended sequences. On the right, the present-clip motion-prior diffusion model uses multimodal causality and temporal interactions to decouple and predict motion states, covering head, lip, and expression movements while maintaining a clear separation of identity and motion features.
• Figure 2: The overview of memory-efficient temporal attention. It can dynamically update and integrate historical motion features with current ones.
• Figure 3: Qualitative comparison on HDTF and CelebV-HQ. Our method achieves the best generation results, particularly in identity consistency and motion detail.
• Figure 4: User study results of identity consistency, motion synchronization, and video quality. Higher values indicate better performance.
• Figure 5: Visualization results and SSIM scores during long-term generation. We find that w/ $F_a$ offers a distinct advantage in maintaining both identity and contextual consistency.