RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer

TL;DR

RAP introduces a hybrid attention mechanism for fine-grained audio control, and a static-dynamic training-inference paradigm that avoids explicit motion supervision, which achieves precise audio-driven control, mitigates long-term temporal drift, and maintains high visual fidelity.

Abstract

Audio-driven portrait animation aims to synthesize realistic and natural talking head videos from an input audio signal and a single reference image. While existing methods achieve high-quality results by leveraging high-dimensional intermediate representations and explicitly modeling motion dynamics, their computational complexity renders them unsuitable for real-time deployment. Real-time inference imposes stringent latency and memory constraints, often necessitating the use of highly compressed latent representations. However, operating in such compact spaces hinders the preservation of fine-grained spatiotemporal details, thereby complicating audio-visual synchronization RAP (Real-time Audio-driven Portrait animation), a unified framework for generating high-quality talking portraits under real-time constraints. Specifically, RAP introduces a hybrid attention mechanism for fine-grained audio control, and a static-dynamic training-inference paradigm that avoids explicit motion supervision. Through these techniques, RAP achieves precise audio-driven control, mitigates long-term temporal drift, and maintains high visual fidelity. Extensive experiments demonstrate that RAP achieves state-of-the-art performance while operating under real-time constraints.

Figures (9)

• Figure 1: Illustration of the proposed portrait animation framework RAP. Given a reference image and an audio clip, the model generates a natural and vivid talking portrait.
• Figure 2: Overview of the proposed RAP framework. (a) Overview of the RAP pipeline, where audio and image inputs are encoded into compressed tokens, followed by DiT-based denoising to generate high-quality portrait videos. (b) The hybrid attention block conducts cross-attention at both short-term and long-term temporal scales, and fuses the results to capture multi-scale audio-motion dependencies. (c) A step-wise inference strategy that progressively guides video generation in the latent space by inheriting information across timesteps.
• Figure 3: Qualitative comparison with existing approaches on HDTF and VFHQ dataset. Videos are available in the Supplementary Material.
• Figure 4: Comparison of temporal consistency and visual drift. Warmer colors denote larger motion amplitude. Our method exhibits minimal background flicker and shift, while preserving significant facial motion.
• Figure 5: Human preferences among RAP and baselines.