Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion

TL;DR

The paper addresses semantic 3D scene completion (SSC) from monocular camera inputs by bridging stereo geometry and BEV representations through a unified occupancy-based framework called BRGScene. It introduces a Mutual Interactive Ensemble (MIE) consisting of a Bi-directional Reliable Interaction (BRI) module and a Dual Volume Ensemble (DVE) to enable pixel-level, reliable fusion between dense stereo and BEV volumes, producing a high-quality ensembled volume $\mathbf{V}_{ens}$. SSC is performed via an outer-product fusion with BEV context and a 3D UNet, trained with depth, semantic, and geometric losses plus class weighting to supervise both geometry and semantics. Experiments on SemanticKITTI show BRGScene surpassing state-of-the-art camera-based SSC methods, with notable gains on small and distant objects and effective hallucination in unseen regions, illustrating the practical potential of stereo-BEV fusion for low-cost 3D scene understanding.

Abstract

3D semantic scene completion (SSC) is an ill-posed perception task that requires inferring a dense 3D scene from limited observations. Previous camera-based methods struggle to predict accurate semantic scenes due to inherent geometric ambiguity and incomplete observations. In this paper, we resort to stereo matching technique and bird's-eye-view (BEV) representation learning to address such issues in SSC. Complementary to each other, stereo matching mitigates geometric ambiguity with epipolar constraint while BEV representation enhances the hallucination ability for invisible regions with global semantic context. However, due to the inherent representation gap between stereo geometry and BEV features, it is non-trivial to bridge them for dense prediction task of SSC. Therefore, we further develop a unified occupancy-based framework dubbed BRGScene, which effectively bridges these two representations with dense 3D volumes for reliable semantic scene completion. Specifically, we design a novel Mutual Interactive Ensemble (MIE) block for pixel-level reliable aggregation of stereo geometry and BEV features. Within the MIE block, a Bi-directional Reliable Interaction (BRI) module, enhanced with confidence re-weighting, is employed to encourage fine-grained interaction through mutual guidance. Besides, a Dual Volume Ensemble (DVE) module is introduced to facilitate complementary aggregation through channel-wise recalibration and multi-group voting. Our method outperforms all published camera-based methods on SemanticKITTI for semantic scene completion. Our code is available on https://github.com/Arlo0o/StereoScene.

Figures (9)

• Figure 1: Overview of the proposed BRGScene. The figure illustrates the stereo inputs, SSC prediction results and ground truth from left to right. We can see that our method shows promising performance in completing semantic scenes, especially for those challenging distant small objects, as indicated by the car highlighted with a red box.
• Figure 2: Overall framework of our proposed BRGScene. Given input stereo images, we employ 2D UNet to extract image features. The BEV latent volume and stereo geometric volume are constructed by a BEV Constructor and a Stereo Constructor, respectively. To bridge their representation gap for fine-grained reliable perception, a Mutual Interactive Ensemble block is proposed to take advantage of complementary merits of the volumes.
• Figure 3: The structure of the proposed Bi-directional Reliable Interaction module, which is designed for pixel-level reliable geometry information interaction.
• Figure 4: The structure of the proposed Dual Volume Ensemble module, which is devised for mutually beneficial aggregation.
• Figure 5: Qualitative results on the SemanticKITTI validation set. The overlay shadow areas at the bottom of semantic predictions denote unseen scenery out of the camera's field of view (FOV).