DepthSSC: Monocular 3D Semantic Scene Completion via Depth-Spatial Alignment and Voxel Adaptation

TL;DR

DepthSSC tackles monocular 3D semantic scene completion by coupling a Spatially-Transformed Graph Fusion to align depth and voxel features with a graph-based fusion, and a Geometrically-aware Voxelization that adapts voxel resolution to local geometry. Building on VoxFormer, it introduces ASAN-based voxel-to-node alignment, graph clustering, and resolution-aware deformable attention to preserve fine structures and boundaries. The two-stage training with occupancy and semantic losses, plus a Hausdorff-based geometric preservation term, yields state-of-the-art results on SemanticKITTI and KITTI-360 datasets and demonstrates robustness to depth noise. Overall, DepthSSC provides a principled approach to retrieving accurate 3D structure from monocular inputs, with strong implications for autonomous driving perception and scene understanding.

Abstract

The task of 3D semantic scene completion using monocular cameras is gaining significant attention in the field of autonomous driving. This task aims to predict the occupancy status and semantic labels of each voxel in a 3D scene from partial image inputs. Despite numerous existing methods, many face challenges such as inaccurately predicting object shapes and misclassifying object boundaries. To address these issues, we propose DepthSSC, an advanced method for semantic scene completion using only monocular cameras. DepthSSC integrates the Spatial Transformation Graph Fusion (ST-GF) module with Geometric-Aware Voxelization (GAV), enabling dynamic adjustment of voxel resolution to accommodate the geometric complexity of 3D space. This ensures precise alignment between spatial and depth information, effectively mitigating issues such as object boundary distortion and incorrect depth perception found in previous methods. Evaluations on the SemanticKITTI and SSCBench-KITTI-360 dataset demonstrate that DepthSSC not only captures intricate 3D structural details effectively but also achieves state-of-the-art performance.

Figures (4)

• Figure 1: Goal of Our Approach. Demonstrates DepthSSC's superiority in handling complex 3D environments for semantic scene completion. Contrasted with VoxFormer, DepthSSC excels in accurately predicting occupancy grids for both nearby and distant objects, overcoming geometric complexity. The highlighted areas showcase VoxFormer's challenges with misrecognition and non-recognition. As illustrated in the first row, VoxFormer fails to adequately utilize depth relationships to distinguish between different vehicles. In the second row, under complex lighting conditions, key features of objects at varying distances are lost, preventing VoxFormer from effectively inferring the 3D structure of the scene. Our DepthSSC addresses this issue by taking into account the geometric complexity of 3D spaces and allowing for dynamic adjustment of voxel resolution.
• Figure 2: Pipeline of the DepthSSC. The process begins with a backbone extracting 2D features from input images, followed by the projection of these 2D features into the 3D voxel domain using deformable self-attention and deformable cross-attention mechanisms. These mechanisms integrate spatial and geometric information. The Spatially-Transformed Graph Fusion (ST-GF) module corrects spatial misalignments by predicting a 3D affine transformation matrix for voxel queries, forming a graph structure that enables feature fusion via Graph Convolutional Networks (GCNs). Geometrically-aware Voxelization (GAV) adapts voxel resolution dynamically based on the geometric complexity of regions, enhancing the model’s ability to represent intricate structures and improve overall 3D scene completion.
• Figure 3: Workflow of the Spatially-Transformed Graph Fusion (ST-GF) Module. The ST-GF module corrects spatial misalignments by predicting a 3D affine transformation matrix $\Theta_{ijk}$. Voxels are then clustered based on spatial similarity and processed through a Graph Convolutional Network (GCN) to aggregate features. Finally, an inverse affine transformation and trilinear interpolation are applied, ensuring accurate spatial alignment and enhancing 3D scene coherence.
• Figure 4: Visualization results against the state-of-the-art monocular SSC methods on SemanticKITTI behley2019semantickitti (val set).