Multi-Resolution Alignment for Voxel Sparsity in Camera-Based 3D Semantic Scene Completion

TL;DR

This work tackles voxel sparsity in camera-based 3D semantic scene completion by introducing Multi-Resolution Alignment (MRA), a framework that enforces self-consistency across multi-resolution 3D features to provide auxiliary supervision beyond sparse voxel labels. It integrates a Multi-resolution View Transformer to project 2D image features into multi-resolution 3D volumes, a Cubic Semantic Anisotropy module to assess voxel significance via semantic reassignment and cubic neighborhood differences, and a Critical Distribution Alignment mechanism with a circulated loss to align critical voxel distributions across resolutions. The approach yields state-of-the-art results on SemanticKITTI and SSCBench-KITTI-360, demonstrates generalization to Surround-nuScenes, and shows favorable efficiency trade-offs given the accuracy gains. Overall, MRA advances robust 3D scene understanding in autonomous driving by leveraging cross-resolution feature alignment to compensate for sparse supervisory signals.

Abstract

Camera-based 3D semantic scene completion (SSC) offers a cost-effective solution for assessing the geometric occupancy and semantic labels of each voxel in the surrounding 3D scene with image inputs, providing a voxel-level scene perception foundation for the perception-prediction-planning autonomous driving systems. Although significant progress has been made in existing methods, their optimization rely solely on the supervision from voxel labels and face the challenge of voxel sparsity as a large portion of voxels in autonomous driving scenarios are empty, which limits both optimization efficiency and model performance. To address this issue, we propose a \textit{Multi-Resolution Alignment (MRA)} approach to mitigate voxel sparsity in camera-based 3D semantic scene completion, which exploits the scene and instance level alignment across multi-resolution 3D features as auxiliary supervision. Specifically, we first propose the Multi-resolution View Transformer module, which projects 2D image features into multi-resolution 3D features and aligns them at the scene level through fusing discriminative seed features. Furthermore, we design the Cubic Semantic Anisotropy module to identify the instance-level semantic significance of each voxel, accounting for the semantic differences of a specific voxel against its neighboring voxels within a cubic area. Finally, we devise a Critical Distribution Alignment module, which selects critical voxels as instance-level anchors with the guidance of cubic semantic anisotropy, and applies a circulated loss for auxiliary supervision on the critical feature distribution consistency across different resolutions. The code is available at https://github.com/PKU-ICST-MIPL/MRA_TIP.

Figures (5)

• Figure 1: (a) Supervision signals from voxel labels suffer from the challenge of voxel sparsity. (b) Our self alignment across multi-resolution 3D feature distributions serve as auxiliary supervision signals for addressing the voxel sparsity.
• Figure 2: The overall architecture of our MRA framework. The Multi-resolution View Transformer (MVT) module projects 2D image features into multi-resolution 3D features and conduct seed feature alignment for the propagation of discriminative semantics to the whole scene. The Cubic Semantic Anisotropy (CSA) module identifies the semantic-ware significance of each voxel with semantic reassignment and aggregation of cubic semantic differences. The Critical Distribution Alignment (CDA) module selects critical voxels with the guidance of CSA and employs the circulated loss as auxiliary supervision on the critical distribution consistency.
• Figure 3: Illustrations of the cubic semantic anisotropy for inner voxels inside the object and corner voxels on the object boundary. A thorough consideration of the surface-, edge-, and vertex-adjacency within the cubic neighborhood is crucial for accurately identifying voxel-wise semantic-aware significance.
• Figure 4: Qualitative visualization results on the SemanticKITTI validation set. Cyan boxes outline the occupancy ground truth. Red boxes indicate false occupancy predictions of the best comparison method SGN, and green boxes indicate the improved scene completion results with more accurate object boundaries generated by our MRA approach. Better viewed when zoomed in.
• Figure 6: Training convergence curves of mIoU and IoU on the SemanticKITTI validation set, comparing MRA and SGN.