Occupancy World Model for Robots

TL;DR

This work tackles indoor robotics by forecasting the evolution of 3D occupancy in a scene. It introduces RoboOccWorld, a two-stage framework that first tokenizes current occupancy with a VQ-VAE and then uses Hybrid Spatio-Temporal Aggregation (HSTA) combined with a Conditional Causal State Attention (CCSA) guided autoregressive transformer to predict future occupancy $O_{future}$ from current occupancy $O_{current}$, history $O_{history}$, and next pose $P$ via $O_{future} = F_{world}(O_{current}, O_{history}, P)$. The method targets indoor scenarios by restructuring the OccWorld-ScanNet benchmark for indoor 3D occupancy evolution prediction and demonstrates clear improvements over outdoor-focused baselines in both next-state and autoregressive prediction tasks. This work advances indoor scene understanding for robotic planning and exploration by enabling more accurate, pose-aware occupancy forecasting, with potential impact on perception, navigation, and decision-making in indoor environments. $O_{current} = F_{mono}(I_{rgb}, E, K)$ and $O_{future} = F_{world}(O_{current}, O_{history}, P)$ are central formulations guiding the approach.

Abstract

Understanding and forecasting the scene evolutions deeply affect the exploration and decision of embodied agents. While traditional methods simulate scene evolutions through trajectory prediction of potential instances, current works use the occupancy world model as a generative framework for describing fine-grained overall scene dynamics. However, existing methods cluster on the outdoor structured road scenes, while ignoring the exploration of forecasting 3D occupancy scene evolutions for robots in indoor scenes. In this work, we explore a new framework for learning the scene evolutions of observed fine-grained occupancy and propose an occupancy world model based on the combined spatio-temporal receptive field and guided autoregressive transformer to forecast the scene evolutions, called RoboOccWorld. We propose the Conditional Causal State Attention (CCSA), which utilizes camera poses of next state as conditions to guide the autoregressive transformer to adapt and understand the indoor robotics scenarios. In order to effectively exploit the spatio-temporal cues from historical observations, Hybrid Spatio-Temporal Aggregation (HSTA) is proposed to obtain the combined spatio-temporal receptive field based on multi-scale spatio-temporal windows. In addition, we restructure the OccWorld-ScanNet benchmark based on local annotations to facilitate the evaluation of the indoor 3D occupancy scene evolution prediction task. Experimental results demonstrate that our RoboOccWorld outperforms state-of-the-art methods in indoor 3D occupancy scene evolution prediction task. The code will be released soon.

Figures (7)

• Figure 1: Given previous 3D occupancy observations and next-step trajectories, our occupancy World model based on the combined spatio-temporal receptive field and guided autoregressive transformer can forecast scene evolutions for robots' decision and exploration.
• Figure 2: The indoor monocular images of different sequences. The red arrows corresponding to the images represent the direction of camera movement.
• Figure 3: The comparison between existing methods and our method on spatio-temporal interaction. The differentcoloredblocks represent different windows and $T$ is the time dimension.
• Figure 4: The Framework of RoboOccWorld. The RoboOccWorld is divided into 2 stages. In the first stage, given the current occupancy $O_{current}$, we employ the VQ-VAE as the tokenizer for occupancy scene representation. In the second stage, we partition the scene tokens into multi-scale spatio-temporal chunks, and utilize the Hybrid Spatio-Temporal Aggregation (HSTA) module that consists of both Long-term (L) and Short-term (S) contexts to effectively aggregate the spatio-temporal information. Subsequently, the Conditional Causal State Attention (CCSA) mechanism utilizes an autoregressive transformer guided by the next-step camera pose to generate forecasts for the next state occupancy.
• Figure 5: The pipeline of the proposed HSTA.