From Binary to Semantic: Utilizing Large-Scale Binary Occupancy Data for 3D Semantic Occupancy Prediction

TL;DR

This paper tackles the high annotation cost of 3D semantic occupancy by leveraging abundant binary occupancy data. It introduces a two-stage framework that first learns from binary occupancy to improve geometric understanding and then refines semantic predictions, plus an offboard auto-labeling approach that uses GT binary occupancy to generate semantic labels. Empirical results show significant gains over baselines on OpenScene and nuScenes, with binary pre-training outperforming traditional pre-training methods, and the offboard auto-labeling pipeline producing high-quality pseudo-labels. The approach enables scalable, cost-effective semantic occupancy prediction for vision-based autonomous driving systems, with practical implications for data collection and labeling workflows.

Abstract

Accurate perception of the surrounding environment is essential for safe autonomous driving. 3D occupancy prediction, which estimates detailed 3D structures of roads, buildings, and other objects, is particularly important for vision-centric autonomous driving systems that do not rely on LiDAR sensors. However, in 3D semantic occupancy prediction -- where each voxel is assigned a semantic label -- annotated LiDAR point clouds are required, making data acquisition costly. In contrast, large-scale binary occupancy data, which only indicate occupied or free space without semantic labels, can be collected at a lower cost. Despite their availability, the potential of leveraging such data remains unexplored. In this study, we investigate the utilization of large-scale binary occupancy data from two perspectives: (1) pre-training and (2) learning-based auto-labeling. We propose a novel binary occupancy-based framework that decomposes the prediction process into binary and semantic occupancy modules, enabling effective use of binary occupancy data. Our experimental results demonstrate that the proposed framework outperforms existing methods in both pre-training and auto-labeling tasks, highlighting its effectiveness in enhancing 3D semantic occupancy prediction. The code is available at https://github.com/ToyotaInfoTech/b2s-occupancy

Figures (16)

• Figure 1: Compared to (a) semantic occupancy prediction, (b) occupancy segmentation utilizes binary occupancy as an additional input along with multi-view images, leading to higher performance.
• Figure 2: Overview of the model architecture. Multi-view features extracted from an image encoder are processed through an image-3D transformation module to obtain a 3D voxel feature representation. Pre-training is performed up to the binary decoder (depicted with solid lines) using $L_{\rm{binary}}$, while fine-tuning is applied to the entire model with both $L_{\rm{binary}}$ and $L_{\rm{semantic}}$. For learning-based auto-labeling, GT binary occupancy is used in place of the binary decoder to identify occupied regions. Red voxels at the center of occupancy images represent the ego car.
• Figure 3: Schematic of Image-to-3D transformation modules. (a) Lift-Splat (LS) and (b) deformable cross-attention modules. The voxel grids represent the 3D voxel representation.
• Figure 4: Schematic of baseline strategies. The voxel grids represent the 3D voxel representation.
• Figure 5: Performance comparison of (a) BEVFormer-based and (b) LS-based methods on OpenScene.