Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation

TL;DR

The paper tackles real-time, controllable portrait animation using diffusion-based video generation, where streaming constraints conflict with high-quality, long-horizon coherence. It introduces Knot Forcing, a streaming autoregressive framework that combines chunk-wise generation with a global reference KV cache, a Temporal Knot that overlaps adjacent chunks, and a running-ahead mechanism to maintain long-term semantic alignment. The approach yields improved visual stability, reduced flicker, and robust identity preservation for infinite sequences on consumer GPUs, with ablations validating the contribution of each component. Compared with both autoregressive and causal diffusion baselines, Knot Forcing achieves competitive or superior quality at similar or lower latency, demonstrating practical viability for interactive applications in real time.

Abstract

Real-time portrait animation is essential for interactive applications such as virtual assistants and live avatars, requiring high visual fidelity, temporal coherence, ultra-low latency, and responsive control from dynamic inputs like reference images and driving signals. While diffusion-based models achieve strong quality, their non-causal nature hinders streaming deployment. Causal autoregressive video generation approaches enable efficient frame-by-frame generation but suffer from error accumulation, motion discontinuities at chunk boundaries, and degraded long-term consistency. In this work, we present a novel streaming framework named Knot Forcing for real-time portrait animation that addresses these challenges through three key designs: (1) a chunk-wise generation strategy with global identity preservation via cached KV states of the reference image and local temporal modeling using sliding window attention; (2) a temporal knot module that overlaps adjacent chunks and propagates spatio-temporal cues via image-to-video conditioning to smooth inter-chunk motion transitions; and (3) A "running ahead" mechanism that dynamically updates the reference frame's temporal coordinate during inference, keeping its semantic context ahead of the current rollout frame to support long-term coherence. Knot Forcing enables high-fidelity, temporally consistent, and interactive portrait animation over infinite sequences, achieving real-time performance with strong visual stability on consumer-grade GPUs.

Figures (10)

• Figure 1: Streaming video generation setup. (a) T2V causal video diffusion. (b) Our approach for portrait animation: given a reference frame, we generate video autoregressively with a short sliding attention window, ensuring low latency, balanced computation, and stable identity preservation.
• Figure 2: The video clips generated by Rolling Forcing, LongLive, and Self Forcing are presented from top to bottom, respectively. Significant temporal artifacts can be observed between adjacent frames, such as inconsistent object motion (first two rows) and abrupt changes in color tone (last row). Zoom in for details.
• Figure 3: Attention masks of different causal designs. IoU of attention contexts between time steps $t$ and $t+1$ is computed to quantify the change in contextual coherence. (a): Causvid and Self Forcing. (b): LongLive. (c): Ours.
• Figure 4: Key components of Knot Forcing. (a) illustrates the proposed temporal knot module. (b) illustrates the rollout inference pipeline with global context running ahead.
• Figure 5: We assess inter-frame dependency by ablating each context frame and computing the L2 difference in attention outputs (relative to the original), normalized by the L2 norm of the unmodified output. The $10^\text{th}$ frame is used as the anchor, the resulting scores indicate each frame’s contribution to the current frame’s attention output.