StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars

TL;DR

This work proposes a two-stage autoregressive adaptation and acceleration framework that applies autoregressive distillation and adversarial refinement to adapt a high-fidelity human video diffusion model for real-time, interactive streaming and develops a one-shot, interactive, human avatar model capable of generating both natural talking and listening behaviors with coherent gestures.

Abstract

Real-time, streaming interactive avatars represent a critical yet challenging goal in digital human research. Although diffusion-based human avatar generation methods achieve remarkable success, their non-causal architecture and high computational costs make them unsuitable for streaming. Moreover, existing interactive approaches are typically limited to head-and-shoulder region, limiting their ability to produce gestures and body motions. To address these challenges, we propose a two-stage autoregressive adaptation and acceleration framework that applies autoregressive distillation and adversarial refinement to adapt a high-fidelity human video diffusion model for real-time, interactive streaming. To ensure long-term stability and consistency, we introduce three key components: a Reference Sink, a Reference-Anchored Positional Re-encoding (RAPR) strategy, and a Consistency-Aware Discriminator. Building on this framework, we develop a one-shot, interactive, human avatar model capable of generating both natural talking and listening behaviors with coherent gestures. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing approaches in generation quality, real-time efficiency, and interaction naturalness. Project page: https://streamavatar.github.io .

Figures (11)

• Figure 1: We propose StreamAvatar, which adapts human video diffusion models for real-time, streaming, and interactive video generation through a two-stage framework. Given a reference image and user/agent audio streams as input, StreamAvatar generates high-resolution streaming human video in real-time, producing vivid talking/listening expressions and gestures.
• Figure 2: Overview of the two-stage autoregressive adaptation and acceleration framework. The original bidirectional DiT is first transformed into a block-causal DiT with block size $C=3$. Then, in stage 1, we apply Score Identity Distillation to distill from the bidirectional teacher into a block-causal student. A Reference Sink and Reference-Anchored Positional Re-encoding is introduced to improve long-term stability and consistency. In stage 2, we apply an adversarial refinement process guided by a Consistency-Aware Discriminator, to further improve generation quality, consistency, and stability.
• Figure 3: Vanilla RoPE vs RoPE with Reference-Anchored Positional Re-encoding (RAPR). RAPR improves long video generation without the need for long video training.
• Figure 4: The architecture of the transformer block from our interactive human generation model. We extend the original block with audio-related attention modules to support talking and listening condition.
• Figure 5: Qualitative comparison with SoTA talking avatar video generation methods. Please zoom in for details.