Toward Memory-Aided World Models: Benchmarking via Spatial Consistency

TL;DR

The paper tackles the problem of maintaining spatial consistency in world models by introducing LoopNav, a Minecraft-based looped-trajectory dataset designed to train memory-enabled agents. It employs an explore-then-generate benchmark that tests long-horizon spatial reconstruction on $A \rightarrow B$ context and $B \rightarrow A$ generation, using metrics like $FVD$, LPIPS, and SSIM. Four baselines (Oasis, Mineworld, DIAMOND, NWM) reveal significant gaps in memory-based spatial coherence, highlighting the need for explicit memory modules and spatial representations. By open-sourcing the dataset, benchmark, and code, the work provides a practical platform to advance memory mechanisms and spatial-structure modeling in generative world models with real open-world complexity.

Abstract

The ability to simulate the world in a spatially consistent manner is a crucial requirements for effective world models. Such a model enables high-quality visual generation, and also ensures the reliability of world models for downstream tasks such as simulation and planning. Designing a memory module is a crucial component for addressing spatial consistency: such a model must not only retain long-horizon observational information, but also enables the construction of explicit or implicit internal spatial representations. However, there are no dataset designed to promote the development of memory modules by explicitly enforcing spatial consistency constraints. Furthermore, most existing benchmarks primarily emphasize visual coherence or generation quality, neglecting the requirement of long-range spatial consistency. To bridge this gap, we construct a dataset and corresponding benchmark by sampling 150 distinct locations within the open-world environment of Minecraft, collecting about 250 hours (20 million frames) of loop-based navigation videos with actions. Our dataset follows a curriculum design of sequence lengths, allowing models to learn spatial consistency on increasingly complex navigation trajectories. Furthermore, our data collection pipeline is easily extensible to new Minecraft environments and modules. Four representative world model baselines are evaluated on our benchmark. Dataset, benchmark, and code are open-sourced to support future research.

Figures (3)

• Figure 1: The Loop-Style Trajectory Data for Training and Benchmarking. To be able to explicitly enforce spatial consistency, our dataset follows a loop-style navigation trajectory, including both ABA and ABCA types. Our benchmark design explicitly separates the generation phase from the exploration phase and only evaluates the video quality of the generation part.
• Figure 2: Overview of the Minecraft Elements. Top row, left to right: Examples of villages, biomes, and structures in Minecraft. Bottom row, left to right: composition of sampling locations in our dataset; a bird-eye view of real ABA-type exploration trajectories (display steps 5, 15, and 30 for simplicity); Simplified Action Space used for data collection and agent interaction.
• Figure 3: Qualitative Result of Four Baselines. Top to buttom: Ground Truth(GT), Oasis, Mineworld(MW), DIAMOND(DIA), Navigation World Model(NWM). Leftmost label “t=...” indicates the start context range accepted by each model. For example, for the Oasis model, “t=52” means frames 52 to 83 are used as context. All models begin rollout from frame 84.