AvatarSync: Rethinking Talking-Head Animation through Phoneme-Guided Autoregressive Perspective

TL;DR

AvatarSync addresses the limitations of diffusion-based talking-head generation—namely inter-frame flicker and slow inference—by adopting a phoneme-guided autoregressive framework. It introduces a two-stage pipeline that decouples semantics from visual dynamics: facial keyframe generation guided by phonemes and timestamps, followed by timestamp-aware interpolation via a selective state-space model. A Phoneme-Frame Causal Attention Mask enforces precise phoneme-to-frame alignment, enabling editable, segment-level control. Evaluations on Chinese CMLR and English HDTF datasets show state-of-the-art visual fidelity, temporal coherence, and cross-lingual generalization, with near-linear inference scaling with phoneme count, supporting real-time applications. The work also discusses ethical considerations and reproducibility, with plans to release data enhancements and code for responsible research use.

Abstract

Talking-head animation focuses on generating realistic facial videos from audio input. Following Generative Adversarial Networks (GANs), diffusion models have become the mainstream, owing to their robust generative capacities. However, inherent limitations of the diffusion process often lead to inter-frame flicker and slow inference, restricting their practical deployment. To address this, we introduce AvatarSync, an autoregressive framework on phoneme representations that generates realistic and controllable talking-head animations from a single reference image, driven directly by text or audio input. To mitigate flicker and ensure continuity, AvatarSync leverages an autoregressive pipeline that enhances temporal modeling. In addition, to ensure controllability, we introduce phonemes, which are the basic units of speech sounds, and construct a many-to-one mapping from text/audio to phonemes, enabling precise phoneme-to-visual alignment. Additionally, to further accelerate inference, we adopt a two-stage generation strategy that decouples semantic modeling from visual dynamics, and incorporate a customized Phoneme-Frame Causal Attention Mask to support multi-step parallel acceleration. Extensive experiments conducted on both Chinese (CMLR) and English (HDTF) datasets demonstrate that AvatarSync outperforms existing talking-head animation methods in visual fidelity, temporal consistency, and computational efficiency, providing a scalable and controllable solution.

Figures (10)

• Figure 1: Comparison of GANs-based, diffusion-based, and our autoregressive method. The left and middle panels summarize key limitations of GANs and diffusion models. The right panel illustrates the advantages of our autoregressive method.
• Figure 2: Inter-frame Flicker Visualization. Left: reference frame; subsequent panels show pixel-wise differences between consecutive frames, where scattered high-difference regions reveal temporal flicker.
• Figure 3: The overall framework of AvatarSync. The pipeline first normalizes text/audio into a compact phoneme token sequence via a many-to-one mapping, and tokenizes the reference image into visual tokens. Next, a two-stage autoregressive generator performs Facial Keyframe Generation under a Phoneme-Frame Causal Attention Mask, then inserts intermediate frames using a timestamp-aware selective that interleaves keyframes for linear-time global context. Finally, the decoder reconstructs RGB frames to animate character.
• Figure 4: Generation Time Comparison. AvatarSync scales nearly linearly with phoneme count, while others exhibit exponential growth. At 20 phonemes, it is 2.4 times faster than Hallo and remains the most efficient.
• Figure 5: Qualitative comparison on the CMLR and HDTF dataset. (a) Top: ground-truth frames. Middle: results from baseline models. Bottom: Each phoneme (represented as pinyin for Chinese) is aligned with its corresponding frame. (b) Inter-frame flicker visualization, where pixel-wise differences between consecutive frames highlight temporal inconsistencies across methods.