PAN: A World Model for General, Interactable, and Long-Horizon World Simulation

TL;DR

PAN introduces Generative Latent Prediction (GLP), a general, interactive world model that unifies latent dynamics grounded in observations with a video diffusion decoder for high-fidelity perception. It combines a vision-language encoder/transformer backbone with a causal diffusion decoder (Causal Swin-DPM) and trains on large-scale video–action data using generative supervision to avoid latent collapse. The approach achieves state-of-the-art performance among open-source baselines and competitiveness with some closed-source systems on action-conditioned world simulation, long-horizon forecasting, and simulative planning, with qualitative demonstrations across diverse domains. This work advances toward general-purpose world models capable of imagined reasoning, counterfactual planning, and real-time, action-conditioned prediction, enabling more capable AI agents. Future work will scale PAN to broader modalities and more intricate interactivity to further improve causal reasoning and planning in complex environments.

Abstract

A world model enables an intelligent agent to imagine, predict, and reason about how the world evolves in response to its actions, and accordingly to plan and strategize. While recent video generation models produce realistic visual sequences, they typically operate in the prompt-to-full-video manner without causal control, interactivity, or long-horizon consistency required for purposeful reasoning. Existing world modeling efforts, on the other hand, often focus on restricted domains (e.g., physical, game, or 3D-scene dynamics) with limited depth and controllability, and struggle to generalize across diverse environments and interaction formats. In this work, we introduce PAN, a general, interactable, and long-horizon world model that predicts future world states through high-quality video simulation conditioned on history and natural language actions. PAN employs the Generative Latent Prediction (GLP) architecture that combines an autoregressive latent dynamics backbone based on a large language model (LLM), which grounds simulation in extensive text-based knowledge and enables conditioning on language-specified actions, with a video diffusion decoder that reconstructs perceptually detailed and temporally coherent visual observations, to achieve a unification between latent space reasoning (imagination) and realizable world dynamics (reality). Trained on large-scale video-action pairs spanning diverse domains, PAN supports open-domain, action-conditioned simulation with coherent, long-term dynamics. Extensive experiments show that PAN achieves strong performance in action-conditioned world simulation, long-horizon forecasting, and simulative reasoning compared to other video generators and world models, taking a step towards general world models that enable predictive simulation of future world states for reasoning and acting.

Figures (10)

• Figure 1: PAN model architecture. It consists of an autoregressive LLM-based world model backbone for long-horizon world simulation, and a video diffusion decoder for observation prediction.
• Figure 2: Transformer block of the video diffusion decoder in PAN.
• Figure 3: Denoising process for Causal Swin-DPM.
• Figure 4: An example of a complete reasoning and planning process. Given an initial state and a goal specification, a VLM proposes candidate actions at each step. The world model simulates each candidate's outcome as a "thought experiment", and the VLM selects the action whose predicted state shows maximal goal progress. This iterative process repeats until task completion or step limit reached. Gray arrows show explored alternatives; blue arrows trace the selected trajectory. The tree-structured search demonstrates how world models enable multi-step planning by simulating hypothetical futures before plan finalization.
• Figure 5: Model performance on Action Simulation Fidelity, Long-Horizon Forecast and Simulative Planning tasks.