RoboOcc: Enhancing the Geometric and Semantic Scene Understanding for Robots

TL;DR

RoboOcc tackles the challenge of 3D occupancy prediction for robots by addressing the underutilization of geometry and opacity in Gaussian-based scene representations. It introduces two key modules: Opacity-guided Self-Encoder (OSE) to reduce semantic ambiguity in overlapping Gaussians, and Geometry-aware Cross-Encoder (GCE) to enable fine-grained geometric modeling, iteratively updating a set of 3D Gaussians and converting them to dense voxels via Gaussian-to-Voxel splatting. The method achieves state-of-the-art results on Occ-ScanNet (local) and EmbodiedOcc-ScanNet (global) with strong ablations validating the contributions; RoboOcc also demonstrates robustness to Gaussian parameter variations. These advances significantly improve autonomous robots' ability to perceive and reason about indoor environments, enabling more reliable navigation and interaction in complex scenes.

Abstract

3D occupancy prediction enables the robots to obtain spatial fine-grained geometry and semantics of the surrounding scene, and has become an essential task for embodied perception. Existing methods based on 3D Gaussians instead of dense voxels do not effectively exploit the geometry and opacity properties of Gaussians, which limits the network's estimation of complex environments and also limits the description of the scene by 3D Gaussians. In this paper, we propose a 3D occupancy prediction method which enhances the geometric and semantic scene understanding for robots, dubbed RoboOcc. It utilizes the Opacity-guided Self-Encoder (OSE) to alleviate the semantic ambiguity of overlapping Gaussians and the Geometry-aware Cross-Encoder (GCE) to accomplish the fine-grained geometric modeling of the surrounding scene. We conduct extensive experiments on Occ-ScanNet and EmbodiedOcc-ScanNet datasets, and our RoboOcc achieves state-of the-art performance in both local and global camera settings. Further, in ablation studies of Gaussian parameters, the proposed RoboOcc outperforms the state-of-the-art methods by a large margin of (8.47, 6.27) in IoU and mIoU metric, respectively. The codes will be released soon.

Figures (6)

• Figure 1: Considering that current Gaussian representations lack effective utilization of geometry and opacity properties, we propose an enhanced geometric and semantic scene understanding 3D occupancy prediction method for robots. Based on this, the robot makes the local occupancy prediction in an indoor scene with accepted monocular RGB and completes the global occupancy prediction through exploration over time.
• Figure 2: The pipeline of Gaussian-to-Voxel splatting. The green, pink, blue ellipsoid and grid represents different 3D Gaussian and voxel in 3D space, respectively. The arrow points from Gaussian center to voxel. The comparison will be conducted under controlled variable conditions.
• Figure 3: The overall framework of the proposed RoboOcc. It consists of Image Encoder, Gaussian Encoder and Gaussian-to-Voxel Splatting. For Image Encoder, We use the EfficientNet to extract multi-scale semantic features from monocular image. We then randomly initialized a set of Gaussian queries and anchors. We use the 3D Gaussians to represent indoor scene and update the Gaussian-based representation based on semantic and structural features extracted from indoor monocular image with Gaussian Encoder. The Gaussian-to-Voxel splatting is finally employed to generate dense 3D occupancy prediction via local aggregation of Gaussians.
• Figure 4: Qualitative Analysis on the Occ-ScanNet-mini dataset. It can be seen that the proposed RoboOcc can model the scene better. It can capture the scene layout and classify various semantic instances more accurately.
• Figure 5: Additional visualizations in scene0028.