MAGIC-Talk: Motion-aware Audio-Driven Talking Face Generation with Customizable Identity Control

TL;DR

MAGIC-Talk addresses the challenge of generating customizable, temporally stable talking-face videos from a single reference image and audio. It introduces ReferenceNet for identity-preserving appearance encoding with text-driven editing and AnimateNet for motion-aware lip-sync, aided by a decoupled cross-attention mechanism and edge contour conditioning. A Variational Motion Generator supplies structured motion priors, and a training-free progressive sampling fusion enables high-quality long-form video generation. Across HDTF and MEAD, MAGIC-Talk demonstrates superior identity preservation, motion realism, and synchronization, highlighting its potential for robust virtual avatars and digital media creation. The work advances controllable, generalizable talking-face synthesis with practical applicability in real-world scenarios.

Abstract

Audio-driven talking face generation has gained significant attention for applications in digital media and virtual avatars. While recent methods improve audio-lip synchronization, they often struggle with temporal consistency, identity preservation, and customization, especially in long video generation. To address these issues, we propose MAGIC-Talk, a one-shot diffusion-based framework for customizable and temporally stable talking face generation. MAGIC-Talk consists of ReferenceNet, which preserves identity and enables fine-grained facial editing via text prompts, and AnimateNet, which enhances motion coherence using structured motion priors. Unlike previous methods requiring multiple reference images or fine-tuning, MAGIC-Talk maintains identity from a single image while ensuring smooth transitions across frames. Additionally, a progressive latent fusion strategy is introduced to improve long-form video quality by reducing motion inconsistencies and flickering. Extensive experiments demonstrate that MAGIC-Talk outperforms state-of-the-art methods in visual quality, identity preservation, and synchronization accuracy, offering a robust solution for talking face generation.

Figures (7)

• Figure 1: Illustration of our proposed MAGIC-Talk framework for customizable and temporally consistent talking face generation. Given a single reference image, speech audio, and a text prompt, our model enables fine-grained control for talking face generation.
• Figure 2: An overview of proposed MAGIC-Talk framework for one-shot, customizable talking face generation. The framework consists of two key components: ReferenceNet, which preserves identity, while enabling fine-grain facial editing through text guidance, and AnimateNet, which maps structured motion priors to enhance temporal coherence and speech-driven dynamics.
• Figure 3: Qualitative comparison of our method with baseline talking face generation approaches. The methods are categorized into three groups: (1) No emotion conditioning, (2) Emotion label or reference video guidance, and (3) Text description guidance.
• Figure 4: Illustration of the ablation study. Depicting the impact of key components in MAGIC-Talk.
• Figure 5: Effect of the motion module on talking face generation.