LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation

TL;DR

This paper addresses real-time multimodal interactive video generation conditioned on text, image, and audio by diagnosing latency and stability challenges in diffusion-based approaches. It develops an improved on-policy distillation recipe that centers on curating high-quality multimodal conditions, converging ODE initialization, and aggressive optimization, enabling a 4-step autoregressive model that matches or exceeds bidirectional baselines with ~20× throughput improvements. The distilled model underpins LiveTalk, a real-time avatar system integrated with a reasoning/speech module and Identity-preserving mechanisms, achieving sub-second latency and superior multi-turn coherence relative to state-of-the-art avatars. Collectively, the work demonstrates practical real-time multimodal human-AI interaction with robust audiovisual synchronization and scalable, interactive video generation.

Abstract

Real-time video generation via diffusion is essential for building general-purpose multimodal interactive AI systems. However, the simultaneous denoising of all video frames with bidirectional attention via an iterative process in diffusion models prevents real-time interaction. While existing distillation methods can make the model autoregressive and reduce sampling steps to mitigate this, they focus primarily on text-to-video generation, leaving the human-AI interaction unnatural and less efficient. This paper targets real-time interactive video diffusion conditioned on a multimodal context, including text, image, and audio, to bridge the gap. Given the observation that the leading on-policy distillation approach Self Forcing encounters challenges (visual artifacts like flickering, black frames, and quality degradation) with multimodal conditioning, we investigate an improved distillation recipe with emphasis on the quality of condition inputs as well as the initialization and schedule for the on-policy optimization. On benchmarks for multimodal-conditioned (audio, image, and text) avatar video generation including HDTF, AVSpeech, and CelebV-HQ, our distilled model matches the visual quality of the full-step, bidirectional baselines of similar or larger size with 20x less inference cost and latency. Further, we integrate our model with audio language models and long-form video inference technique Anchor-Heavy Identity Sinks to build LiveTalk, a real-time multimodal interactive avatar system. System-level evaluation on our curated multi-turn interaction benchmark shows LiveTalk outperforms state-of-the-art models (Sora2, Veo3) in multi-turn video coherence and content quality, while reducing response latency from 1 to 2 minutes to real-time generation, enabling seamless human-AI multimodal interaction.

Figures (3)

• Figure 2: Degraded training performance with Self Forcing DMD. Left and middle columns show failure cases from Self Forcing DMD training, exhibiting quality degradation. Right column shows stable results from our method.
• Figure 3: Examples of multimodal-conditioned avatar video generation by our model. Our model generates temporally coherent video with natural facial expressions, accurate lip-sync to the audio conditions, and consistent visual identity across frames.
• Figure 4: Ablation study visualization. Generated video shows progressive improvements for each ablated component: (1) curated multimodal conditions, (2) ODE initialization for full convergence (20k steps), and (3) aggressive hyperparameter settings (doubled learning rates, CFG=6).