Enhancing 3D Semantic Scene Completion with a Refinement Module

TL;DR

This work introduces ESSC-RM, a unified refinement framework that enhances 3D semantic scene completion by refining coarse voxel predictions with a model-agnostic 3D U-Net backbone. It integrates Progressive Neighborhood Attention Module (PNAM) for multiscale voxel reasoning and a Visual–Language Guidance Module (VLGM) to inject text-derived priors via Text–Voxel Fusion blocks. The approach demonstrates consistent semantic gains on SemanticKITTI when applied to strong backbones like CGFormer and weaker baselines like MonoScene, while analyzing efficiency and qualitative improvements. By combining geometric refinement with cross-modal priors, ESSC-RM offers a practical, plug-and-play path to improve SSC across heterogeneous architectures. The results suggest promising directions for cross-modal priors and efficient refinement in large-scale 3D scene understanding tasks.

Abstract

We propose ESSC-RM, a plug-and-play Enhancing framework for Semantic Scene Completion with a Refinement Module, which can be seamlessly integrated into existing SSC models. ESSC-RM operates in two phases: a baseline SSC network first produces a coarse voxel prediction, which is subsequently refined by a 3D U-Net-based Prediction Noise-Aware Module (PNAM) and Voxel-level Local Geometry Module (VLGM) under multiscale supervision. Experiments on SemanticKITTI show that ESSC-RM consistently improves semantic prediction performance. When integrated into CGFormer and MonoScene, the mean IoU increases from 16.87% to 17.27% and from 11.08% to 11.51%, respectively. These results demonstrate that ESSC-RM serves as a general refinement framework applicable to a wide range of SSC models.

Figures (8)

• Figure 1: Overall architecture of ESSC-RM. An SSC backbone first predicts a coarse semantic voxel grid from image and/or LiDAR input. The refinement module embeds the discrete voxel labels into continuous features and processes them with a 3D U-Net enhanced by PNAM and VLGM, finally producing a refined semantic volume $\hat{\mathbf{Y}}^{\prime}$.
• Figure 2: Three-dimensional U-shaped neural network (3D U-Net) backbone used in the refinement module. The encoder consists of stacked feature encoding blocks (FEBs) that progressively downsample the input and extract multi-scale features, a bottleneck aggregates global context, and the decoder consists of stacked feature aggregation blocks (FABs) that progressively upsample the features and predict semantic voxel outputs at four spatial scales.
• Figure 3: Core components of the three-dimensional U-shaped neural network (3D U-Net). The feature encoding block (FEB) applies three-dimensional convolutions, normalization, and residual connections to encode geometric and semantic information, while the feature aggregation block (FAB) upsamples low-resolution features and fuses them with skip connections to recover fine spatial details.
• Figure 4: Progressive Neighborhood Attention U-Net (PNA U-Net) 2023_PNA. PNAM replaces the feature aggregation blocks (FABs) at intermediate scales ($1{:}2$, $1{:}4$, $1{:}8$) with attention-based FABs, while the finest-scale FAB remains convolutional for efficiency.
• Figure 5: Core components of the PNA-based Feature Aggregation Block (FAB). Each block combines self-attention (SA) and neighborhood cross-attention (NCA) to jointly model long-range dependencies and local geometric consistency.