DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model

TL;DR

DyStream tackles the challenge of real-time, dyadic talking-head generation by introducing a flow-matching autoregressive model that processes dual-track audio with a causal encoder and a short lookahead, achieving frame latency around 34 ms and overall system latency under 100 ms. The approach couples a Motion-aware Autoencoder with a Flow-matching AR generator and uses a sliding-window, CFG-guided inference to produce frame-by-frame motion conditioned on both speaker and listener audio. It further optimizes real-time performance with a customized Wav2Vec2 encoder that supports a 60 ms lookahead while maintaining causality, and demonstrates state-of-the-art lip-sync on HDTF offline and online metrics (8.13 and 7.61, respectively). The method enables interactive, visually embodied agents capable of both speaking and listening in streaming environments, with broad implications for education, sales, and virtual companionship.

Abstract

Generating realistic, dyadic talking head video requires ultra-low latency. Existing chunk-based methods require full non-causal context windows, introducing significant delays. This high latency critically prevents the immediate, non-verbal feedback required for a realistic listener. To address this, we present DyStream, a flow matching-based autoregressive model that could generate video in real-time from both speaker and listener audio. Our method contains two key designs: (1) we adopt a stream-friendly autoregressive framework with flow-matching heads for probabilistic modeling, and (2) We propose a causal encoder enhanced by a lookahead module to incorporate short future context (e.g., 60 ms) to improve quality while maintaining low latency. Our analysis shows this simple-and-effective method significantly surpass alternative causal strategies, including distillation and generative encoder. Extensive experiments show that DyStream could generate video within 34 ms per frame, guaranteeing the entire system latency remains under 100 ms. Besides, it achieves state-of-the-art lip-sync quality, with offline and online LipSync Confidence scores of 8.13 and 7.61 on HDTF, respectively. The model, weights and codes are available.

Figures (5)

• Figure 1: System pipeline. DyStream generates talking-head videos from a single reference image and dyadic stream. First, an Autoencoder disentangles the reference image into a static appearance feature $\mathbf{v}_{app}$ and an initial, identity-agnostic motion feature $\mathbf{m}^0$. Next, the Audio-to-Motion Generator takes the initial motion $\mathbf{m}^0$ and the audio stream as input to generate a new sequence of audio-aligned motion features $\mathbf{m}^{1:N}$. Finally, the Autoencoder's decoder synthesizes the output video by warping the appearance feature $\mathbf{v}_{app}$ according to the generated motion sequence $\mathbf{m}^{1:N}$.
• Figure 2: The architecture of our audio-to-motion generator. Our model comprises two core modules: an autoregressive network (blue) and a flow matching head (orange). The autoregressive network, built from causal self-attention and MLP blocks, processes the audio, anchor, and previous motion inputs to generate a conditioning signal $c^N$. This signal is fed into the flow matching head, a stack of MLPs and AdaLN layers. Here, it is injected via AdaLN to guide a multi-step flow matching process to produce the final motion $m^N$. Finally, the newly generated motion $m^N$ is used to warp the reference image into the output frame, while simultaneously being fed back into the autoregressive network as input for the subsequent generation.
• Figure 3: Effect of classifier-free guidance (CFG) on lipsync performance. We evaluate Sync-C across a grid of CFG All and CFG Speaker.
• Figure 4: Ablation Study on Anchor Frame Selection Strategy. Visual comparison of long-sequence generation results using the same input audio and initial reference image. The rows from top to bottom correspond to three different anchor selection strategies used during training: our method (anchor sampled from the last 10 frames), a random anchor, and no anchor.
• Figure 5: Qualitative results and limitations.Top: Given reference images from different views, our approach can generate realistic videos that maintain consistent head pose over time. Bottom: Our method have the limitation when hands are overlapped with face.