READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation

TL;DR

This paper tackles the real-time constraint of diffusion-based audio-driven talking head generation by introducing READ, a diffusion-transformer framework that compresses video latents with a Temporal VAE, aligns audio-visual signals via a pre-trained SpeechAE, and synthesizes latents with an Audio-to-Video Diffusion Transformer. A novel Asynchronous Noise Scheduler (ANS) enables long-term, temporally consistent generation by combining asynchronous forward noise with motion-guided reverse diffusion. Experiments on HDTF and MEAD show that READ achieves competitive visual quality and lip-sync while delivering substantially faster backbone runtimes than state-of-the-art methods, including robust performance for extended generation. The work validates the practical potential of end-to-end, real-time diffusion for expressive video synthesis and highlights directions to further improve motion quality and fidelity in challenging regions.

Abstract

The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation.

Figures (8)

• Figure 1: The framework of READ. During training, we first pre-train the SpeechAE for speech feature temporal compression, shown in (b). Then we train the total framework using the asynchronous forward process, shown in (c). During inference, we conduct the asynchronous motion-guided reverse process by ANS, also shown in (c).
• Figure 2: Ablation results of ANS on HDTF dataset.
• Figure 3: Trade-off between performance and runtime under different inference steps on HDTF dataset.
• Figure 4: Case study of talking head generation methods.
• Figure 5: Detailed architecture of SpeechAE.