TesserAct: Learning 4D Embodied World Models

TL;DR

TesserActadvanced learns a 4D embodied world model by training on RGB-DN videos to predict temporally and spatially coherent 4D scenes. It leverages latent diffusion on RGB-DN streams, coupled with depth optimization from normals, and reconstructs 4D scenes for robust action planning. The work introduces a 4D embodied video dataset, a CogVideoX-based architecture, and novel temporal-spatial consistency losses, showing improved 4D scene prediction and downstream RLBench task performance over 2D video-based baselines. This framework enables more realistic offline policy learning and planning through imagined rollouts in a 4D geometric world representation.

Abstract

This paper presents an effective approach for learning novel 4D embodied world models, which predict the dynamic evolution of 3D scenes over time in response to an embodied agent's actions, providing both spatial and temporal consistency. We propose to learn a 4D world model by training on RGB-DN (RGB, Depth, and Normal) videos. This not only surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal changes into their predictions, but also allows us to effectively learn accurate inverse dynamic models for an embodied agent. Specifically, we first extend existing robotic manipulation video datasets with depth and normal information leveraging off-the-shelf models. Next, we fine-tune a video generation model on this annotated dataset, which jointly predicts RGB-DN (RGB, Depth, and Normal) for each frame. We then present an algorithm to directly convert generated RGB, Depth, and Normal videos into a high-quality 4D scene of the world. Our method ensures temporal and spatial coherence in 4D scene predictions from embodied scenarios, enables novel view synthesis for embodied environments, and facilitates policy learning that significantly outperforms those derived from prior video-based world models.

Figures (12)

• Figure 1: We propose TesserAct, the 4D Embodied World Model, which takes an input image and text instruction to generate RGB, depth, and normal videos, reconstructing a 4D scene and predicting actions. Our model not only achieves strong performance on in-domain data (right) but also generalizes effectively to unseen scenes, novel objects (top left), and cross-domain scenarios (bottom left).
• Figure 2: Architecture and Training Overview of TesserAct.
• Figure 3: Effect of consistency and regularization loss on 4D scene reconstruction. The red boxes highlight the inconsistent regions.
• Figure 4: Qualitative results of (a) In-domain 4D generation results. (b) Generalization over unseen scenes and objects. (c) Novel view synthesis.
• Figure 5: Visualization of the optimized 3D robotic scene reconstruction using our method. The untextured renderings show enhanced detail (green box) and improved surface smoothness (red box). The side perspective view highlights accurate shape and geometry optimization, including the perpendicular alignment of the wall and table (red boxes).