ET-Former: Efficient Triplane Deformable Attention for 3D Semantic Scene Completion From Monocular Camera

TL;DR

This work tackles semantic scene completion from a single monocular image by introducing ET-Former, which combines a triplane deformable attention mechanism with a Conditional Variational Autoencoder to jointly predict a semantic occupancy map and its uncertainty. The method operates in two stages: first producing occupancy queries via a triplane-based attention module to guide 3D understanding, then generating the semantic map conditioned on the RGB image while quantifying uncertainty through a CVAE latent space. Key contributions include a memory-efficient triplane-deformable decoder, a staged training strategy with CVAE-based uncertainty, and substantial performance gains on SemanticKITTI (IoU up from $44.71$ to $51.49$, mIoU up from $15.04$ to $16.30$) with remarkably lower GPU memory usage ($10.9$ GB). The dual outputs (semantic map and uncertainty) enable safer navigation and risk-aware decision making in real-world autonomous systems.

Abstract

We introduce ET-Former, a novel end-to-end algorithm for semantic scene completion using a single monocular camera. Our approach generates a semantic occupancy map from single RGB observation while simultaneously providing uncertainty estimates for semantic predictions. By designing a triplane-based deformable attention mechanism, our approach improves geometric understanding of the scene than other SOTA approaches and reduces noise in semantic predictions. Additionally, through the use of a Conditional Variational AutoEncoder (CVAE), we estimate the uncertainties of these predictions. The generated semantic and uncertainty maps will help formulate navigation strategies that facilitate safe and permissible decision making in the future. Evaluated on the Semantic-KITTI dataset, ET-Former achieves the highest Intersection over Union (IoU) and mean IoU (mIoU) scores while maintaining the lowest GPU memory usage, surpassing state-of-the-art (SOTA) methods. It improves the SOTA scores of IoU from 44.71 to 51.49 and mIoU from 15.04 to 16.30 on SeamnticKITTI test, with a notably low training memory consumption of 10.9 GB. Project page: https://github.com/jingGM/ET-Former.git.

Figures (7)

• Figure 1: Our ET-Former predicts the semantic occupancy map from a monocular camera image to complete the scene and estimate the uncertainties of the occupancy map.
• Figure 2: Overall Architecture: We present a two-stage pipeline for processing mono-cam images and generate both a semantic occupancy map $\mathbf{m}_s$ and its corresponding uncertainty map $\mathbf{m}_u$. In stage 1, we introduce a novel triplane-based deformable attention model to generate the occupancy queries $\mathbf{m}_o$ from the given mono-cam images, which reduces high-dimensional 3D feature processing to 2D computations. The detailed artchitecture of stage 1 is shown in Figure \ref{['fig:stage1']}. In stage 2, we employ the efficient triplane-based deformable attention mechanism (detailed in Figure \ref{['fig:stage2']}) to generate the semantic map, with the inferred voxels from stage 1 as input and conditioned on the RGB image. To estimate the uncertainty in the semantic map, we incorporate a CVAE method, and quantify the uncertainty using the variance of the CVAE latent samples.
• Figure 3: (a) Deformable Triplane Decoder of Stage 2: We propose triplane-based efficient self-deformable attention (ESDA) and efficient cross-deformable attention (ECDA) methods in the decoder to generate 3D features. The input to the decoder consists of Gaussian samples $\hat{\mathbf{f}}_o$ and image features $\mathbf{f}_r$. The Gaussian samples are projected onto three orthogonal planes based on positional embeddings. The three planes features are completed by ESDA and then enhanced with image features, $\mathbf{f}_r$, by ECDA. Finally, the plane features are concatenated and upsampled to generate the semantic occupancy map. (b) Detailed Structure of ECDA: ECDA utilizes image features at deformed points $(\mathbf{p}_i + \Delta \mathbf{p}_i)$ and completed plane feature $\mathbf{f}_{\text{self}}$ to conduct cross attention.
• Figure 4: Semantic Qualitative Results: The blue rectangles highlight areas that are present in the Ground Truth but are missing from both VoxFormer and MonoScene predictions. The red rectangles marked the areas with wrong semantic estimations from Voxformer and MonoScene.
• Figure 5: Uncertainty Qualitative Results: In column 4, blue voxels show low uncertainty areas, while red voxels, highlighted by yellow rectangles, indicate high uncertainty. These uncertain areas are typically outside the camera's FOV, or obscured by vegetation (row 1) or buildings (rows 2-3).