Walk through Paintings: Egocentric World Models from Internet Priors

TL;DR

The paper addresses the challenge of action-conditioned visual world modeling by repurposing pretrained video diffusion models and introducing lightweight action conditioning that plugs into the models’ timestep conditioning, enabling controllable, high-resolution future predictions across diverse embodiments. By training only a small action-conditioning module atop the fixed backbone, EgoWM achieves strong cross-embodiment generalization (from 3-DoF to 25-DoF) and can operate in non-realistic domains such as paintings, while significantly improving inference latency over prior baselines. A key contribution is the Structural Consistency Score (SCS), which measures whether stable scene elements evolve coherently with actions, decoupled from perceptual realism. The approach demonstrates improved action-following, robust generalization to unseen environments, and practical real-time viability for navigation and manipulation tasks, highlighting a scalable path toward general-purpose visual dynamics models.

Abstract

What if a video generation model could not only imagine a plausible future, but the correct one, accurately reflecting how the world changes with each action? We address this question by presenting the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model, enabling controllable future prediction. Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers. This allows the model to follow actions faithfully while preserving realism and strong generalization. Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids, where predicting egocentric joint-angle-driven dynamics is substantially more challenging. The model produces coherent rollouts for both navigation and manipulation tasks, requiring only modest fine-tuning. To evaluate physical correctness independently of visual appearance, we introduce the Structural Consistency Score (SCS), which measures whether stable scene elements evolve consistently with the provided actions. EgoWM improves SCS by up to 80 percent over prior state-of-the-art navigation world models, while achieving up to six times lower inference latency and robust generalization to unseen environments, including navigation inside paintings.

Figures (10)

• Figure 1: Our framework generates future frame predictions (shown in blue) that accurately follow the provided robot actions (ground-truth frames shown in green). Notably, it effortlessly generalizes to vastly different embodiments, tasks and domains, including non-realistic ones, such as navigating within paintings (top right). This generalization is enabled by our simple, universal architecture.
• Figure 2: Our method embeds action sequences or arbitrary dimensionality into a universal feature space and injects these embeddings into a pre-trained video diffusion model by reusing its timestep-conditioning pathway (shown in the top center). This enables turning any passive video generation model into a world model without destroying the pre-trained representation.
• Figure 3: Each column shows two generated rollouts compared to the corresponding ground-truth frame, with their LPIPS, DreamSim, and SCS scores. Perceptual metrics incorrectly favor the visually sharper but physically inconsistent sample, while SCS correctly identifies the sequence that follows the true action trajectory, accurately capturing structural consistency.
• Figure 4: Qualitative results on ego-centric navigation. Both variants of our model generate realistic, temporally coherent sequences that accurately follow the provided action trajectories, while NWM exhibits noticeable drift (e.g., veering right in the first column).
• Figure 5: The inference latency of autoregressive NWM is much higher compared to SVD and Cosmos, and the gap becomes more pronounced as the number of frames increases.