Learning 3D Persistent Embodied World Models

TL;DR

This work tackles the instability of long-horizon video-based world models by introducing a 3D persistent memory that stores previously generated content as a volumetric map. By encoding actions into relative camera transforms via Plücker embeddings and generating RGB-D futures conditioned on a 3D map, the model maintains geometric and semantic consistency across extended sequences, including unseen regions. The approach demonstrates superior video fidelity and persistence over baselines and enables practical embodied AI capabilities such as planning with model-predictive control, trajectory ranking, and policy learning in new environments. Overall, grounding video-based world models in a 3D memory framework advances the reliability and transferability of learned dynamics in complex, partially observable environments.

Abstract

The ability to simulate the effects of future actions on the world is a crucial ability of intelligent embodied agents, enabling agents to anticipate the effects of their actions and make plans accordingly. While a large body of existing work has explored how to construct such world models using video models, they are often myopic in nature, without any memory of a scene not captured by currently observed images, preventing agents from making consistent long-horizon plans in complex environments where many parts of the scene are partially observed. We introduce a new persistent embodied world model with an explicit memory of previously generated content, enabling much more consistent long-horizon simulation. During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent. This generation is then aggregated into a persistent 3D map of the environment. By conditioning the video model on this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world. Finally, we illustrate the efficacy of such a world model in downstream embodied applications, enabling effective planning and policy learning.

Figures (10)

• Figure 1: 3D Persistent Video Generation. Given the context video that defines the top-left 3D scene, the baseline world model deviates from this layout and introduces contradictory elements. In contrast, with 3D memory, our approach preserves observed structures and generates content consistent with the original context.
• Figure 2: Overview of our framework. (a) Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video. (d) The memory is incrementally updated after each video generation. (b, c) We train the memory blocks only and freeze DiT blocks in the second training stage.
• Figure 3: Qualitative Comparison of Ours with baselines. Given 3D memory and camera trajectory, the videos generated by our model are high-quality and closely match the ground truth, while NWM bar2024navigationworldmodels, without the memory mechanism, generates new contents that conflict with the ground truth. We use green boxes to show consistency and red boxes to show conflict.
• Figure 4: Qualitative Results of Consistency Generation. We autoregressively generate videos four times and present the final images from each generation. Our model can generate consistent content after revisiting the same location.
• Figure 5: Qualitative Examples of Long Video Generation. Our model can generate long videos with memory.