From Masks to Worlds: A Hitchhiker's Guide to World Models

TL;DR

This work articulates a narrow, five-stage roadmap to true world models built from three core subsystems: a Generative Heart for world dynamics, an Interactive Loop for real-time perception-action, and a Memory System for long-horizon coherence. It traces the evolution from Stage I mask-based self-supervision to Stage II unified architectures, Stage III interactive generative models, Stage IV externalized memory and consistency, and finally Stage V the threshold of persistence, agency, and emergence. The paper argues that true world models must integrate generation, interaction, and memory into a self-sustaining ecosystem, addressing coherence, compression, and alignment as central challenges. It further positions such living worlds as powerful scientific instruments for studying complex adaptive systems, beyond entertainment or training benchmarks. The practical impact lies in guiding research toward durable, agent-inhabited worlds with memory and emergent dynamics that can be used to explore and understand complex phenomena.

Abstract

This is not a typical survey of world models; it is a guide for those who want to build worlds. We do not aim to catalog every paper that has ever mentioned a ``world model". Instead, we follow one clear road: from early masked models that unified representation learning across modalities, to unified architectures that share a single paradigm, then to interactive generative models that close the action-perception loop, and finally to memory-augmented systems that sustain consistent worlds over time. We bypass loosely related branches to focus on the core: the generative heart, the interactive loop, and the memory system. We show that this is the most promising path towards true world models.

Figures (2)

• Figure 1: The evolution of world models across five stages.
• Figure 2: The architecture of a true world model.