VividAnimator: An End-to-End Audio and Pose-driven Half-Body Human Animation Framework

TL;DR

Vivid Animator tackles the challenge of high-fidelity, half-body humanoid video synthesis driven by audio and sparse hand poses from a single reference image. It introduces three core components: a Hand Clarity Codebook (HCC) pretrained offline to inject rich hand priors, a Dual-Stream Audio-Aware Module (DSAA) that decouples head-motion rhythm from lip synchronization, and a Pose Calibration Trick (PCT) to align driving poses with the reference. The framework is built on a latent diffusion model with cross-attention to maintain identity and temporal coherence, achieving superior hand detail, gesture realism, and audio–visual synchronization, as demonstrated by extensive quantitative metrics and qualitative results. These contributions advance controllable, high-fidelity video synthesis with practical implications for entertainment, virtual characters, and film production.

Abstract

Existing for audio- and pose-driven human animation methods often struggle with stiff head movements and blurry hands, primarily due to the weak correlation between audio and head movements and the structural complexity of hands. To address these issues, we propose VividAnimator, an end-to-end framework for generating high-quality, half-body human animations driven by audio and sparse hand pose conditions. Our framework introduces three key innovations. First, to overcome the instability and high cost of online codebook training, we pre-train a Hand Clarity Codebook (HCC) that encodes rich, high-fidelity hand texture priors, significantly mitigating hand degradation. Second, we design a Dual-Stream Audio-Aware Module (DSAA) to model lip synchronization and natural head pose dynamics separately while enabling interaction. Third, we introduce a Pose Calibration Trick (PCT) that refines and aligns pose conditions by relaxing rigid constraints, ensuring smooth and natural gesture transitions. Extensive experiments demonstrate that Vivid Animator achieves state-of-the-art performance, producing videos with superior hand detail, gesture realism, and identity consistency, validated by both quantitative metrics and qualitative evaluations.

Figures (6)

• Figure 1: a) Vivid Animator synthesizes high-fidelity, identity-preserving videos with rich and expressive motion. Given a reference image, a sequence of poses, and audio, our method showcases its capability to produce: b) precise hand reconstruction and c) dynamic, natural head motion synchronized with speech.
• Figure 2: Overview of Vivid Animator, a pipeline for realistic half-body human animation. Given a reference image, an audio clip, and a hand pose sequence as input, our method synthesizes high-fidelity videos with natural motion. A Hand Clarity Codebook (HCC) is introduced to preserve fine-grained hand textures, while a Dual-Stream Audio-Aware Module (DSAA) enables rhythmic and coherent head motion.
• Figure 3: Overview of the Hand Clarity Codebook (HCC). a) The encoder discretizes cropped hand images into latent codes, and the generator reconstructs hands from these embeddings. b) Reconstructions with a 16×16 latent size preserve finer textures and structural details compared to 8×8.
• Figure 4: The results of VividAnimator compared with other audio-driven baselines.
• Figure 5: The results of VividAnimator compared with other pose-driven baselines.