SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction

TL;DR

SelfOcc tackles self-supervised 3D occupancy prediction from video by learning a BEV/TPV-based 3D representation and modeling occupancy as a signed distance field $S$. It employs differentiable volume rendering to supervise 3D geometry, color, and occupancy using multi-view and temporal cues, and introduces an MVS-embedded depth objective to robustly optimize $S$ across multiple depth proposals. The approach achieves state-of-the-art results among vision-based/self-supervised methods on Occ3D-nuScenes and SemanticKITTI, while delivering strong novel depth synthesis and depth estimation performance across nuScenes, KITTI-2015, and SemanticKITTI. By eliminating the need for 3D annotations, SelfOcc enables scalable, perception-focused learning for surround-view autonomous driving with practical impact for robustness and deployment.

Abstract

3D occupancy prediction is an important task for the robustness of vision-centric autonomous driving, which aims to predict whether each point is occupied in the surrounding 3D space. Existing methods usually require 3D occupancy labels to produce meaningful results. However, it is very laborious to annotate the occupancy status of each voxel. In this paper, we propose SelfOcc to explore a self-supervised way to learn 3D occupancy using only video sequences. We first transform the images into the 3D space (e.g., bird's eye view) to obtain 3D representation of the scene. We directly impose constraints on the 3D representations by treating them as signed distance fields. We can then render 2D images of previous and future frames as self-supervision signals to learn the 3D representations. We propose an MVS-embedded strategy to directly optimize the SDF-induced weights with multiple depth proposals. Our SelfOcc outperforms the previous best method SceneRF by 58.7% using a single frame as input on SemanticKITTI and is the first self-supervised work that produces reasonable 3D occupancy for surround cameras on nuScenes. SelfOcc produces high-quality depth and achieves state-of-the-art results on novel depth synthesis, monocular depth estimation, and surround-view depth estimation on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code: https://github.com/huang-yh/SelfOcc.

Figures (12)

• Figure 1: Trained with only video sequences as supervision, our model can predict meaningful geometry for the scene given surround-camera RGB images, which can be further extended to semantic occupancy prediction if 2D segmentation maps are available e.g. from an off-the-shelf segmentor. This task is challenging because it completely depends on video sequences to reconstruct scenes without any 3D supervision. We observe that our model can produce dense and consistent occupancy prediction and even infer the back side of cars.
• Figure 2: Our MVS-embedded strategy effectively enlarges the receptive field of the depth optimization process across the whole epipolar line, which provides a good depth prior (Raw Argmin).
• Figure 3: Framework of the proposed SelfOcc for self-supervised vision-based 3D occupancy prediction. We employ an image backbone and a 3D encoder to generate 3D representations as in BEVFormer li2022bevformer or TPVFormer huang2023tri. To render a novel view, we apply a lightweight MLP on the 3D features to predict the SDF values, color and semantic vectors. We then perform volume rendering to synthesize color, depth and semantic views. We use simple 0-thresholding to predict the occupancy volume.
• Figure 4: Visualization of the effect of different loss functions. The occupancy quality gradually improves with $L_{dep}$, $L_H$ and $L_s$.
• Figure 5: Visualization results for 3D occupancy prediction on the Occ3D-nuScenes tian2023occ3d dataset. Our method achieves comparable visualization quality with ground truth for both semantic and geometric occupancy prediction tasks.