LIVE: Long-horizon Interactive Video World Modeling

TL;DR

Long-horizon autoregressive video world modeling suffers from accumulating errors due to exposure bias. The authors introduce LIVE, a cycle-consistency diffusion framework that forward-rollouts from ground-truth prompts and then reverse-generates to recover the initial state, with diffusion loss applied to the recovered terminal state to bound error propagation and avoid teacher distillation. They unify existing training paradigms (Teacher Forcing and Diffusion Forcing) under a progressive curriculum that modulates the ground-truth versus rollout ratio, enabling stable end-to-end diffusion training. Empirical results on RealEstate10K, UE Engine, and Minecraft demonstrate state-of-the-art long-horizon performance and robust generalization, highlighting LIVE’s capacity to generate stable, high-quality video far beyond training rollout lengths.

Abstract

Autoregressive video world models predict future visual observations conditioned on actions. While effective over short horizons, these models often struggle with long-horizon generation, as small prediction errors accumulate over time. Prior methods alleviate this by introducing pre-trained teacher models and sequence-level distribution matching, which incur additional computational cost and fail to prevent error propagation beyond the training horizon. In this work, we propose LIVE, a Long-horizon Interactive Video world modEl that enforces bounded error accumulation via a novel cycle-consistency objective, thereby eliminating the need for teacher-based distillation. Specifically, LIVE first performs a forward rollout from ground-truth frames and then applies a reverse generation process to reconstruct the initial state. The diffusion loss is subsequently computed on the reconstructed terminal state, providing an explicit constraint on long-horizon error propagation. Moreover, we provide an unified view that encompasses different approaches and introduce progressive training curriculum to stabilize training. Experiments demonstrate that LIVE achieves state-of-the-art performance on long-horizon benchmarks, generating stable, high-quality videos far beyond training rollout lengths.

Figures (21)

• Figure 1: Comparison of autoregressive training paradigms. Teacher Forcing (TF) uses ground truth context during training, causing train-inference mismatch. Diffusion Forcing (DF) injects noise but fails to model real rollout errors. Self-Forcing (SF) employs sequence-level distillation with unbounded error accumulation. Our LIVE performs forward rollout then reverse recovery with frame-level diffusion loss, bounding errors through the cycle-consistency objective.
• Figure 2: Rollout from GT produces semantically diverse content, making direct supervision infeasible. LIVE addresses this by requiring the model to generate back toward the original GT, enabling valid supervision through the cycle-consistency objective.
• Figure 3: LIVE training pipeline. Forward rollout (Left, frozen): Given $p$ prompt frames $x^i$, the model generates the remaining $T-p$ frames $\tilde{x}^j$ via causal attention. Cycle-consistency objective (Right, trainable): The rollout is reversed and used as context to recover the original prompt frames via frame-level diffusion loss, employing reverse attention (right mask, shown for $p=2$).
• Figure 4: Post-training performance from a converged DF checkpoint. Continued DF training stagnates with oscillating metrics, while LIVE achieves substantial improvements that amplify at longer horizons. LIVE converges to comparable FID across 128-frame and 200-frame generation, demonstrating uniform quality regardless of rollout length.
• Figure 5: Progressive training curriculum by increasing rollout ratio. From left to right, as $p$ decreases, more generated frames enter the context, increasing the model's error tolerance while maintaining recoverability through the cycle-consistency objective.