Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

TL;DR

Avatar Forcing introduces a real-time interactive head avatar framework powered by diffusion forcing in a motion latent space to achieve reactive, expressive behavior from multimodal user cues. A Dual Motion Encoder fuses user motion/audio with avatar audio, and a blockwise causal diffusion-forcing generator enables low-latency (< 1 s) generation with KV caching. A label-free enhancement via Direct Preference Optimization tunes the system toward more engaging interactions by comparing preferred and less-preferred motion latents without extra annotations. Evaluations on RealTalk and ViCo rlhg show substantial gains in reactiveness, motion richness, and user preference over state-of-the-art baselines, including dyadic and listening-head models, while maintaining lip-sync and visual quality. The approach advances practical, natural, real-time human–AI conversation with flexible extension potentials, such as additional sensors or controllability cues.

Abstract

Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation. However, current models do not yet convey the feeling of truly interactive communication, often generating one-way responses that lack emotional engagement. We identify two key challenges toward truly interactive avatars: generating motion in real-time under causal constraints and learning expressive, vibrant reactions without additional labeled data. To address these challenges, we propose Avatar Forcing, a new framework for interactive head avatar generation that models real-time user-avatar interactions through diffusion forcing. This design allows the avatar to process real-time multimodal inputs, including the user's audio and motion, with low latency for instant reactions to both verbal and non-verbal cues such as speech, nods, and laughter. Furthermore, we introduce a direct preference optimization method that leverages synthetic losing samples constructed by dropping user conditions, enabling label-free learning of expressive interaction. Experimental results demonstrate that our framework enables real-time interaction with low latency (approximately 500ms), achieving 6.8X speedup compared to the baseline, and produces reactive and expressive avatar motion, which is preferred over 80% against the baseline.

Figures (19)

• Figure 1: Quantitative comparison results on RealTalk realtalk. Best results highlighted in bold. $^{*}$ denotes the reproduced version that is publicly unavailable. We also report the results from a non-interactive talking head model float, shown in gray, for reference.
• Figure 2: Overall architecture of Avatar Forcing. We encode the use motion and audio, as well as avatar audio into a unified condition by Dual Motion Encoder. Causal Motion Generator infer the motion latent block of the avatar, which are then decoded into an avatar video.
• Figure 2: Comparison with talking head generation models on the HDTF hdtf dataset. Second-best results are underlined.
• Figure 3: Architecture of$v_{\theta}$. The look-ahead causal attention mask enables a smooth transition across the blocks.
• Figure 4: Architectural comparison between bidirectional and causal structure. (a) Bidirectional DiT used in INFP infp requires access to the entire temporal window for motion generation. (b) Our blockwise causal DFoT predicts the next block without using future context and supports KV caching.