SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation

TL;DR

SyncAnimation tackles real-time, audio-driven avatar synthesis by introducing a unified NeRF-based framework that jointly renders upper-body and head motion with synchronized lip animation. It integrates AudioPose Syncer, AudioEmotion Syncer, and a High-Synchronization Human Renderer to ensure stable pose-audio alignment, expressive facial dynamics, and seamless torso-head coherence, all while enabling one-shot and zero-shot inferences. The approach uses pose offsets, diversity and stability conditioning, CVAEs for expressions, and a multi-plane hash-based head renderer to achieve high fidelity, natural blinking, and accurate lip-sync, with an emphasis on real-time performance (41 FPS on a RTX 4090). Extensive quantitative and qualitative evaluations show superior image quality, lip-sync accuracy, and motion diversity compared with GAN-, NeRF-, and SD-based baselines, and ablations confirm the necessity of each module. The work advances practical audio-driven avatars by delivering real-time, end-to-end generation of coherent upper-body and facial dynamics from monocular or noisy inputs, suitable for live streaming and conferencing applications.

Abstract

Generating talking avatar driven by audio remains a significant challenge. Existing methods typically require high computational costs and often lack sufficient facial detail and realism, making them unsuitable for applications that demand high real-time performance and visual quality. Additionally, while some methods can synchronize lip movement, they still face issues with consistency between facial expressions and upper body movement, particularly during silent periods. In this paper, we introduce SyncAnimation, the first NeRF-based method that achieves audio-driven, stable, and real-time generation of speaking avatar by combining generalized audio-to-pose matching and audio-to-expression synchronization. By integrating AudioPose Syncer and AudioEmotion Syncer, SyncAnimation achieves high-precision poses and expression generation, progressively producing audio-synchronized upper body, head, and lip shapes. Furthermore, the High-Synchronization Human Renderer ensures seamless integration of the head and upper body, and achieves audio-sync lip. The project page can be found at https://syncanimation.github.io

Figures (7)

• Figure 1: SyncAnimation is the first NeRF-based jointly generative approach that utilizes audio-driven generation to create expressions and an adjustable upper body (left). SyncAnimation requires only audio and monocular, or even noise, to render highly detailed identity information, along with realistic and dynamic facial and upper-body changes, while maintaining audio consistency (right).
• Figure 2: Illustration of SyncAnimation Framework: Given a single image and audio input, the preprocessing stage extracts 3DMM parameters in NeRF space as references for Audio2Pose and Audio2Emotion (or alternatively, noise). The framework then progressively generates the upper body, head, and final lip refinement. Audio2Pose ensures poses consistency with the audio for upper-body generation, while Audio2Emotion aligns facial expression with the audio for head rendering, in conjunction with the generated poses.
• Figure 3: Audio2Pose is designed to reconstruct stable head pose offsets ($\mathbf{Off}\text{ }(\mathbf{e}$) and $\mathbf{Off}\text{ }(\mathbf{t})$) using audio and monocular input. Where, Wav2attr, a pre-trained audio encoder, is employed to encode audio vectors containing character-specific information. Additionally, a Gaussian-based VAE is integrated to introduce a diversity template $\mathbf{S}_{\text{pose}}$, while a stability model $\mathbf{D}_{\text{pose}}$ is implemented based on the poses labels with high dropout rate ($\text{DP}=0.6$), improving the effectiveness of pose reconstruction.
• Figure 4: Audio2Emotion is designed to learn and reconstruct 3DMM expression offsets ($\mathbf{Off}\text{ }(\mathbf{b})$) using audio and monocular input. The structure is similar to that of Audio2Pose, but due to the weak correlation between facial expressions and audio, and the periodic nature of blinking, we modify the diversity template $\mathbf{S}_{\text{exp}}$. It is replaced by a conditional VAE guided by periodic time features $\mathbf{T}_{\tau}$ and context-dependent audio features $\mathbf{A}_{\text{seq}}$.
• Figure 5: Visual comparison with outputs of baselines. GAN-based and NeRF-based methods generate avatar with fixed poses and expressions, while SD-based methods introduce expression changes but lack face detail and pose movement. SyncAnimation uniquely achieves jointly generative, audio-driven realistic expressions and movable poses.