Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving

TL;DR

OccFiner tackles the challenge of unreliable vision-based SSC by introducing an offboard two-stage pipeline that first compensates onboard prediction biases through multi-to-multi local propagation and then enforces long-range consistency via region-centric global propagation with sensor-aware voting. The DualFlow4D transformer enables spatiotemporal refinement in BEV space, while explicit multi-view registration and frustum-aware weighting correct measurement biases, yielding state-of-the-art results on SemanticKITTI and competitive LiDAR-based SSC performance using purely vision-based data. The framework enables automatic SSC annotation, data loop-closure, and city-scale Semantic SSC maps, highlighting the potential of offboard processing to enhance perception in autonomous driving without prohibitive onboard compute. These results suggest significant practical impact for scalable, camera-based 3D scene understanding and map generation in real-world driving scenarios, with ongoing opportunities to improve cross-model generalization and long-range accuracy.

Abstract

Vision-based occupancy prediction, also known as 3D Semantic Scene Completion (SSC), presents a significant challenge in computer vision. Previous methods, confined to onboard processing, struggle with simultaneous geometric and semantic estimation, continuity across varying viewpoints, and single-view occlusion. Our paper introduces OccFiner, a novel offboard framework designed to enhance the accuracy of vision-based occupancy predictions. OccFiner operates in two hybrid phases: 1) a multi-to-multi local propagation network that implicitly aligns and processes multiple local frames for correcting onboard model errors and consistently enhancing occupancy accuracy across all distances. 2) the region-centric global propagation, focuses on refining labels using explicit multi-view geometry and integrating sensor bias, particularly for increasing the accuracy of distant occupied voxels. Extensive experiments demonstrate that OccFiner improves both geometric and semantic accuracy across various types of coarse occupancy, setting a new state-of-the-art performance on the SemanticKITTI dataset. Notably, OccFiner significantly boosts the performance of vision-based SSC models, achieving accuracy levels competitive with established LiDAR-based onboard SSC methods. Furthermore, OccFiner is the first to achieve automatic annotation of SSC in a purely vision-based approach. Quantitative experiments prove that OccFiner successfully facilitates occupancy data loop-closure in autonomous driving. Additionally, we quantitatively and qualitatively validate the superiority of the offboard approach on city-level SSC static maps. The source code will be made publicly available at https://github.com/MasterHow/OccFiner.

Figures (7)

• Figure 1: Current onboard methods generate unreliable occupancy predictions that are inconsistent across different viewpoints. In contrast, our offboard framework constructs a unified and multi-view consistent occupancy map with higher accuracy.
• Figure 2: Overview of the proposed multi-to-multi local propagation network (OccFiner Stage 1). It accepts multiple onboard predictions and relative coordinates as input (multi-input) and simultaneously generates refined predictions for multiple frames (multi-output). This network adeptly executes error compensation and facilitates implicit local propagation, improving SSC quality and temporal consistency across various distances.
• Figure 3: Our proposed DualFlow4D transformer block. It engages spatiotemporal propagation within BEV space and vanilla attention to pillar tokens. This dual approach enables effective matching and flow of semantic and geometric cues for comprehensive scene understanding.
• Figure 4: Overview of the OccFiner two-stage pipeline. Stage 1 performs Multi-to-Multi Local Propagation to refine onboard SSC predictions ($X^T$) into $\hat{Y}^T$. Stage 2 then performs Region-centric Global Propagation on $\hat{Y}^T$, leveraging multi-view geometry and sensor-aware voting to generate the final consistent and accurate offboard SSC map, which can be further applied to auto labeling or data loop-closure.
• Figure 5: Qualitative comparisons. Our offboard solution effectively fixes critical errors in onboard component li2023voxformer, such as large areas of missing road and vehicles. Moreover, OccFiner elevates the performance of pure visual solutions beyond the classic LiDAR-based onboard method SSCNet-full song2017semantic.