Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

Rui Zhang, Yixiao Fang, Zhengnan Lu, Pei Cheng, Zebiao Huang, Bin Fu

TL;DR

LinguaLinker tackles multilingual audio-driven portrait animation by introducing a diffusion-based framework that implicitly controls region-specific facial motion through AdaLN-conditioned gates. The architecture fuses reference-image features with audio embeddings via cross-attention in a denoising network, enabling lip-sync accuracy and temporal coherence without relying on intermediate 3D representations. A two-stage training pipeline and a curated, multilingual dataset support robust performance across diverse portraits and languages, with evaluation on HDTF and CelebV-HQ demonstrating competitive fidelity and synchronization. The approach provides practical versatility for animating any portrait in multiple languages, though it faces diffusion-inference latency and occasional artifact challenges that motivate future improvements.

Abstract

This study delves into the intricacies of synchronizing facial dynamics with multilingual audio inputs, focusing on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. Diverging from traditional parametric models for facial animation, our approach, termed LinguaLinker, adopts a holistic diffusion-based framework that integrates audio-driven visual synthesis to enhance the synergy between auditory stimuli and visual responses. We process audio features separately and derive the corresponding control gates, which implicitly govern the movements in the mouth, eyes, and head, irrespective of the portrait's origin. The advanced audio-driven visual synthesis mechanism provides nuanced control but keeps the compatibility of output video and input audio, allowing for a more tailored and effective portrayal of distinct personas across different languages. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the appropriate motion variations achieved by our method render it a versatile tool for animating any portrait in any language.

LinguaLinker: Audio-Driven Portraits Animation with Implicit Facial Control Enhancement

TL;DR

LinguaLinker tackles multilingual audio-driven portrait animation by introducing a diffusion-based framework that implicitly controls region-specific facial motion through AdaLN-conditioned gates. The architecture fuses reference-image features with audio embeddings via cross-attention in a denoising network, enabling lip-sync accuracy and temporal coherence without relying on intermediate 3D representations. A two-stage training pipeline and a curated, multilingual dataset support robust performance across diverse portraits and languages, with evaluation on HDTF and CelebV-HQ demonstrating competitive fidelity and synchronization. The approach provides practical versatility for animating any portrait in multiple languages, though it faces diffusion-inference latency and occasional artifact challenges that motivate future improvements.

Abstract

This study delves into the intricacies of synchronizing facial dynamics with multilingual audio inputs, focusing on the creation of visually compelling, time-synchronized animations through diffusion-based techniques. Diverging from traditional parametric models for facial animation, our approach, termed LinguaLinker, adopts a holistic diffusion-based framework that integrates audio-driven visual synthesis to enhance the synergy between auditory stimuli and visual responses. We process audio features separately and derive the corresponding control gates, which implicitly govern the movements in the mouth, eyes, and head, irrespective of the portrait's origin. The advanced audio-driven visual synthesis mechanism provides nuanced control but keeps the compatibility of output video and input audio, allowing for a more tailored and effective portrayal of distinct personas across different languages. The significant improvements in the fidelity of animated portraits, the accuracy of lip-syncing, and the appropriate motion variations achieved by our method render it a versatile tool for animating any portrait in any language.
Paper Structure (10 sections, 5 equations, 6 figures, 3 tables)

This paper contains 10 sections, 5 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Architecture Design
  5. Training and Inference
  6. Experiment
  7. Inplementation Details
  8. Evaluation
  9. Results Comparison and Analysis
  10. Conclusion

Figures (6)

  • Figure 1: The proposed method LinguaLinker accepts multilingual audio signals and arbitrary portrait images and generates the corresponding video output.
  • Figure 2: The overview of architecture.
  • Figure 3: More generated results of diverse reference portraits and different languages.
  • Figure 4: Web-sourced video data statistics.
  • Figure 5: The comparison between different methods and the ground truth videos are sampled from CelebV-HQ dataset.
  • ...and 1 more figures