Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control

Renjie Lu, Xulong Zhang, Xiaoyang Qu, Jianzong Wang, Shangfei Wang

TL;DR

MirrorTalk tackles the challenge of preserving a speaker's unique style while achieving accurate lip-sync in talking-face synthesis. It introduces a Semantically-Disentangled Style Encoder (SDSE) trained via a two-stage strategy to isolate style from semantic content, and a spatial-temporal hierarchical modulation for a diffusion-based generator to fuse audio and style across facial regions. Key contributions include the cross-modal semantic alignment with memory banks, HSIC-based disentanglement and a triplet loss, and a region-aware conditioning scheme that adaptively weighs style and audio signals. Experiments on VoxCeleb2, HDTF, and CREMA-D show improved lip-sync accuracy and personalization preservation over state-of-the-art methods, indicating strong potential for personalized avatars in real-time synthesis and digital communication.

Abstract

Synthesizing personalized talking faces that uphold and highlight a speaker's unique style while maintaining lip-sync accuracy remains a significant challenge. A primary limitation of existing approaches is the intrinsic confounding of speaker-specific talking style and semantic content within facial motions, which prevents the faithful transfer of a speaker's unique persona to arbitrary speech. In this paper, we propose MirrorTalk, a generative framework based on a conditional diffusion model, combined with a Semantically-Disentangled Style Encoder (SDSE) that can distill pure style representations from a brief reference video. To effectively utilize this representation, we further introduce a hierarchical modulation strategy within the diffusion process. This mechanism guides the synthesis by dynamically balancing the contributions of audio and style features across distinct facial regions, ensuring both precise lip-sync accuracy and expressive full-face dynamics. Extensive experiments demonstrate that MirrorTalk achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.

MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control

TL;DR

MirrorTalk tackles the challenge of preserving a speaker's unique style while achieving accurate lip-sync in talking-face synthesis. It introduces a Semantically-Disentangled Style Encoder (SDSE) trained via a two-stage strategy to isolate style from semantic content, and a spatial-temporal hierarchical modulation for a diffusion-based generator to fuse audio and style across facial regions. Key contributions include the cross-modal semantic alignment with memory banks, HSIC-based disentanglement and a triplet loss, and a region-aware conditioning scheme that adaptively weighs style and audio signals. Experiments on VoxCeleb2, HDTF, and CREMA-D show improved lip-sync accuracy and personalization preservation over state-of-the-art methods, indicating strong potential for personalized avatars in real-time synthesis and digital communication.

Abstract

Synthesizing personalized talking faces that uphold and highlight a speaker's unique style while maintaining lip-sync accuracy remains a significant challenge. A primary limitation of existing approaches is the intrinsic confounding of speaker-specific talking style and semantic content within facial motions, which prevents the faithful transfer of a speaker's unique persona to arbitrary speech. In this paper, we propose MirrorTalk, a generative framework based on a conditional diffusion model, combined with a Semantically-Disentangled Style Encoder (SDSE) that can distill pure style representations from a brief reference video. To effectively utilize this representation, we further introduce a hierarchical modulation strategy within the diffusion process. This mechanism guides the synthesis by dynamically balancing the contributions of audio and style features across distinct facial regions, ensuring both precise lip-sync accuracy and expressive full-face dynamics. Extensive experiments demonstrate that MirrorTalk achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.
Paper Structure (10 sections, 9 equations, 2 figures, 2 tables)

This paper contains 10 sections, 9 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Facial geometry estimation
  4. Disentangled Framework for Style Embedder
  5. Hierarchical Conditioning for Motion Synthesis
  6. EXPERIMENT
  7. Datasets
  8. Experiment setup
  9. Evaluation
  10. Conclusion

Figures (2)

  • Figure 1: Architecture of MirrorTalk. We first introduce a two-stage training framework (b) to obtain the Semantically-disentangled Style Encoder (SDSE) for talking style prediction. In the main generation pipeline (a), audio and reference video inputs are encoded into compressed tokens as conditions for a diffusion transformer (DiT) model. During the denoising process, we employ a hierarchical modulation strategy (c) to dynamically balances the contributions of audio and style features for distinct facial regions at each timestep $T$. Finally, a Neural Renderer pirenderer utilizes the portrait image and generated motion sequence $P_{1:T}$ to synthesize the final video frames.
  • Figure 2: Qualitative comparsion with AniTalker anitalker, SadTalker sadtalker, Echomimic echomimic and V-Express vexpress. Red and blue boxes highlight incorrect lip movements and facial expressions in the synthesized image respectively. Our methods not only generates accurate lip movements, but also preserves speaker's talking style and facial dynamics.