Terra: Explorable Native 3D World Model with Point Latents

TL;DR

Terra addresses the shortcoming of pixel-aligned world models by proposing a native 3D world model built on a sparse latent point representation. It introduces a point-to-Gaussian VAE (P2G-VAE) to encode input 3D scenes into latent point clouds and decode them into 3D Gaussian primitives, paired with a sparse point flow matching (SPFlow) model to learn a joint geometry-texture distribution via diffusion on latent points. The framework enables explorable world modeling through progressive outpainting in the point latent space, supported by robust position perturbation, adaptive upsampling/refinement, explicit color supervision, and distance-aware trajectory smoothing implemented with flow matching. Evaluations on ScanNet v2 show Terra achieving state-of-the-art reconstruction and generation performance with strong 3D consistency and the ability to render from any viewpoint in a single generation pass, enabling efficient and coherent exploration of large-scale 3D environments.

Abstract

World models have garnered increasing attention for comprehensive modeling of the real world. However, most existing methods still rely on pixel-aligned representations as the basis for world evolution, neglecting the inherent 3D nature of the physical world. This could undermine the 3D consistency and diminish the modeling efficiency of world models. In this paper, we present Terra, a native 3D world model that represents and generates explorable environments in an intrinsic 3D latent space. Specifically, we propose a novel point-to-Gaussian variational autoencoder (P2G-VAE) that encodes 3D inputs into a latent point representation, which is subsequently decoded as 3D Gaussian primitives to jointly model geometry and appearance. We then introduce a sparse point flow matching network (SPFlow) for generating the latent point representation, which simultaneously denoises the positions and features of the point latents. Our Terra enables exact multi-view consistency with native 3D representation and architecture, and supports flexible rendering from any viewpoint with only a single generation process. Furthermore, Terra achieves explorable world modeling through progressive generation in the point latent space. We conduct extensive experiments on the challenging indoor scenes from ScanNet v2. Terra achieves state-of-the-art performance in both reconstruction and generation with high 3D consistency.

Figures (7)

• Figure 1: Method overview. Unlike conventional world models with pixel-aligned representations, we propose Terra as a native 3D world model that describes and generates 3D environments with point latents. Starting with a glimpse of the environment, Terra progressively explores the unknown regions to produce a coherent and complete world simulation.
• Figure 2: Overall pipeline. Terra consists of a point-to-Gaussian VAE and a sparse point flow matching model. The P2G-VAE effectively learns the transformation from input RGB point cloud to point latents, and then to 3D Gaussian primitives. The SPFlow learns the joint distribution of geometry and appearance. Both P2G-VAE and SPFlow adopt native sparse 3D architectures.
• Figure 3: Method details. LAP, Pos. Res., Feat. Res. and NN. denote linear assignment problem, position residual, feature residual and nearest neighbor, respectively.
• Figure 4: Visualization for reconstruction. Terra achieves photorealistic rendering quality for RGB and depth, and learns to complete the partial objects caused by the sensor failure in dark regions.
• Figure 5: Visualization for unconditional generation. Only Terra is able to generate diverse and reasonable scenes while Prometheus and Trellis lack consistent geometry and texture, respectively.