OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flow

TL;DR

OccFlowNet demonstrates that 3D semantic occupancy can be learned with 2D supervision by leveraging differentiable volumetric rendering in a NeRF-inspired framework. It combines a BEVStereo-based 2D-to-3D encoder, a 3D decoder, and rendering-based losses, augmented by temporal rendering and occupancy flow to handle dynamics. The approach achieves state-of-the-art results on Occ3D-nuScenes with 2D supervision and further improves when 2D and 3D labels are combined, highlighting strong potential for self-supervised occupancy estimation. This work bridges the gap between 2D supervision and dense 3D occupancy, and points to future directions such as fully self-supervised depth/semantic cues and learned occupancy flow without 3D bounding boxes.

Abstract

Semantic occupancy has recently gained significant traction as a prominent 3D scene representation. However, most existing methods rely on large and costly datasets with fine-grained 3D voxel labels for training, which limits their practicality and scalability, increasing the need for self-monitored learning in this domain. In this work, we present a novel approach to occupancy estimation inspired by neural radiance field (NeRF) using only 2D labels, which are considerably easier to acquire. In particular, we employ differentiable volumetric rendering to predict depth and semantic maps and train a 3D network based on 2D supervision only. To enhance geometric accuracy and increase the supervisory signal, we introduce temporal rendering of adjacent time steps. Additionally, we introduce occupancy flow as a mechanism to handle dynamic objects in the scene and ensure their temporal consistency. Through extensive experimentation we demonstrate that 2D supervision only is sufficient to achieve state-of-the-art performance compared to methods using 3D labels, while outperforming concurrent 2D approaches. When combining 2D supervision with 3D labels, temporal rendering and occupancy flow we outperform all previous occupancy estimation models significantly. We conclude that the proposed rendering supervision and occupancy flow advances occupancy estimation and further bridges the gap towards self-supervised learning in this domain.

Figures (6)

• Figure 1: Complete architecture of the proposed model. Multi-view images are used as an input to generate density and semantic predictions in a predefined voxel grid. The model is trained by rendering 2D depth and semantic maps from its predictions and by computing a loss on depth and semantic ground truth obtained from LiDAR scans (\ref{['sec:selfsupervised']}). Optionally, the model can be trained using 3D voxel labels (\ref{['sec:supervised']}). N, H, W denote the number of images, the image height and the image width and X, Y, Z represent the voxel grid dimensions. D, C are the latent dimension size and number of classes, respectively.
• Figure 2: Qualitative results on the Occ3D-nuScenes dataset, viewed from the top. The proposed model can estimate static and dynamic objects in 3D correctly and generalizes well to unseen areas. The shown results are generated using the Ours Flow 2D model.
• Figure A.1: Qualitative results on the Occ3D-nuScenes validation set. Each column shows an input image, and the corresponding rendered depth$\hat{D}$ and rendered semantics$\hat{S}$ when rendering the occupancy predictions using volume rendering, simulating the training process. Below, the 2D training labels are shown, generated by projecting annotated LiDAR point scans onto the input images.
• Figure A.2: Qualitative results on the Occ3D-nuScenes validation set. Images of the occupancy space are taken from above and behind the ego vehicle ("third-person" view). The model can estimate the semantic occupancy well compared to the ground truth.
• Figure B.3: Illustration of the dynamic ray filter. Shown are the input image, LiDAR depth and LiDAR semantics for a single frame across three consecutive time steps when using the dynamic ray filtering (\ref{['sec:ray_filter']}). LiDAR points corresponding to dynamic objects are removed in adjacent time steps $t-1$ and $t+1$. In the current time step $t$, points of dynamic objects are preserved.