One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception

TL;DR

The paper addresses the challenge of obtaining reliable 3D volumetric representations for scene perception tasks like MVS and SSC, where single-step methods struggle under occlusions and complex lighting. It proposes Volumetric Probability Diffusion (VPD), a multi-step generative framework that progressively refines volumetric probabilities using a diffusion process conditioned on coarse priors and multi-scale contextual features, aided by the Confidence-Aware Contextual Collaboration (CACC) and Online Filtering (OF) strategies. The approach yields state-of-the-art results on standard MVS benchmarks and demonstrates notable gains in SSC, including surpassing LiDAR-based methods on SemanticKITTI using only camera inputs. This work highlights the potential of diffusion-based, multi-step distribution modeling to produce more accurate and reliable 3D scene representations, with practical impact on robotics and autonomous systems.

Abstract

Numerous studies have investigated the pivotal role of reliable 3D volume representation in scene perception tasks, such as multi-view stereo (MVS) and semantic scene completion (SSC). They typically construct 3D probability volumes directly with geometric correspondence, attempting to fully address the scene perception tasks in a single forward pass. However, such a single-step solution makes it hard to learn accurate and convincing volumetric probability, especially in challenging regions like unexpected occlusions and complicated light reflections. Therefore, this paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps to facilitate fine and reliable scene perception. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD, dedicated to progressively refining the Volumetric Probability in a Diffusion process. Extensive experiments are conducted on scene perception tasks including multi-view stereo (MVS) and semantic scene completion (SSC), to validate the efficacy of our method in learning reliable volumetric representations. Notably, for the SSC task, our work stands out as the first to surpass LiDAR-based methods on the SemanticKITTI dataset.

Figures (7)

• Figure 1: Comparison between single-step approximation and multi-step modeling for 3D scene perception tasks including multi-view stereo (MVS) and semantic scene completion (SSC). We demonstrate the qualitative results of these two methods. The multi-step modeling yields significantly more accurate and reliable results.
• Figure 2: Overall framework of the proposed volumetric probability diffusion (VPD). Given input images, We first extract multi-scale contextual features $\textbf{F}_{context}^{i}$ and coarse probabilistic volumes $\textbf{V}_{prob}$ with off-the-shelf scene perception baselines. Then, $\textbf{V}_{prob}$ concatenated with the random noisy volume ${\boldsymbol{y}}_t$ as input is fed into the 3D diffusion UNet for refinement, while $\textbf{F}_{context}^{i}$ are employed as conditions in CACC to continuously refine the depth volume $\textbf{V}_{depth}^{i}$ in the 3D UNet. Following an iterative process, we progressively estimate a refined volume $\tilde{\boldsymbol{y}}_0$ over multiple steps with diffusion. The estimated volumes are finally fed to the task-specific head to generate depth maps for MVS or occupancy grids for SSC.
• Figure 3: Visualization results in the confidence-aware contextual collaboration (CACC) module. The confidence map and the uncertainty map illustrate the regions with poor estimation, which are effectively refined with CACC.
• Figure 4: Qualitative results for MVS on DTU test set (left two columns) and BlendedMVS validation set (right two columns). Our approach consistently generates more complete predictions in low-texture regions, as well as more accurate and fine-grained results in thin-structure regions.
• Figure 5: Qualitative results for SSC on SemanticKITTI validation set. The shadow areas denote unseen scenery out of the camera’s field of view. Our proposed VPD improves the performance of the baseline in challenging regions.