A Mechanistic View on Video Generation as World Models: State and Dynamics

TL;DR

This work proposes a novel taxonomy centered on two pillars: State Construction and Dynamics Modeling, which categorize state construction into implicit paradigms (context management) and explicit paradigms (latent compression), while dynamics modeling is analyzed through knowledge integration and architectural reformulation.

Abstract

Large-scale video generation models have demonstrated emergent physical coherence, positioning them as potential world models. However, a gap remains between contemporary "stateless" video architectures and classic state-centric world model theories. This work bridges this gap by proposing a novel taxonomy centered on two pillars: State Construction and Dynamics Modeling. We categorize state construction into implicit paradigms (context management) and explicit paradigms (latent compression), while dynamics modeling is analyzed through knowledge integration and architectural reformulation. Furthermore, we advocate for a transition in evaluation from visual fidelity to functional benchmarks, testing physical persistence and causal reasoning. We conclude by identifying two critical frontiers: enhancing persistence via data-driven memory and compressed fidelity, and advancing causality through latent factor decoupling and reasoning-prior integration. By addressing these challenges, the field can evolve from generating visually plausible videos to building robust, general-purpose world simulators.

Figures (5)

• Figure 1: Overview of the paper structure following the mindmap "From Video Generation Model to World Model".
• Figure 2: World Model Inference Cycle.(1) Estimation: Maps historical observations $O_{1:t}$ to a latent representation $S_t$ (Eq. \ref{['eq:state_recurrent']}). (2) Transition: Predicts the future state $S_{t+1}$ given current state and action $A_t$, facilitating mental rollouts and future prediction (Eq. \ref{['eq:implicit_dynamics']}).
• Figure 3: Learning Paradigms: Coupled vs. Decoupled Training.(a) Coupled Training (Closed-loop): The world and policy models are optimized jointly via shared gradients. This often takes place in Unified Architectures or Sequential Architectures (distinct modules with continuous gradient flow). (b) Decoupled Training (Open-loop): The world model serves as a pre-trained, frozen simulator. It enables policy planning but remains fixed, receiving no gradient updates from the action model.
• Figure 4: Functional Primitives of the Memory Mechanism. To manage the computational constraints of long-term video generation, the implicit state is governed by three operations: (a) Compression mitigates bottlenecks by condensing the raw observation history $O_{1:t}$ into compact representations. (b) Retrieval prioritizes contextual relevance by selectively accessing specific historical segments via internal routing or external matching. (c) Consolidation follows a constant computational strategy by dynamically updating the memory buffer, integrating newly generated observations $O_{t+1}$, and evicting obsolete history for continuous streaming.
• Figure 5: Explicit State Architectures. Instead of buffering raw history, these models maintain a compact variable $S_t$. (a) Coupled: State transition is fused within the generative backbone, where $S_t$ evolves as internal hidden activations or weights. (b) Decoupled: Dynamics are structurally separated; a standalone transition model updates $S_t$ before feeding it into the generator.