BUOL: A Bottom-Up Framework with Occupancy-aware Lifting for Panoptic 3D Scene Reconstruction From A Single Image

TL;DR

BUOL presents a bottom-up approach for panoptic 3D scene reconstruction from a single image by replacing the conventional top-down 2D-to-3D lifting with occupancy-aware lifting and deterministic semantic channel mapping. The method predicts rich 2D priors, lifts them into full 3D space using multi-plane occupancy, and refines to dense 3D occupancy and 3D panoptic labels, guided by 2D instance centers for 3D grouping. This design mitigates instance-channel ambiguity and voxel-reconstruction ambiguity, yielding substantial PRQ gains on 3D-Front (+11.81%) and Matterport3D (+7.46%) over prior state-of-the-art methods. The results demonstrate strong generalization to real-world data and reveal the practical benefit of bottom-up panoptic 3D reconstruction for single-view scenes.

Abstract

Understanding and modeling the 3D scene from a single image is a practical problem. A recent advance proposes a panoptic 3D scene reconstruction task that performs both 3D reconstruction and 3D panoptic segmentation from a single image. Although having made substantial progress, recent works only focus on top-down approaches that fill 2D instances into 3D voxels according to estimated depth, which hinders their performance by two ambiguities. (1) instance-channel ambiguity: The variable ids of instances in each scene lead to ambiguity during filling voxel channels with 2D information, confusing the following 3D refinement. (2) voxel-reconstruction ambiguity: 2D-to-3D lifting with estimated single view depth only propagates 2D information onto the surface of 3D regions, leading to ambiguity during the reconstruction of regions behind the frontal view surface. In this paper, we propose BUOL, a Bottom-Up framework with Occupancy-aware Lifting to address the two issues for panoptic 3D scene reconstruction from a single image. For instance-channel ambiguity, a bottom-up framework lifts 2D information to 3D voxels based on deterministic semantic assignments rather than arbitrary instance id assignments. The 3D voxels are then refined and grouped into 3D instances according to the predicted 2D instance centers. For voxel-reconstruction ambiguity, the estimated multi-plane occupancy is leveraged together with depth to fill the whole regions of things and stuff. Our method shows a tremendous performance advantage over state-of-the-art methods on synthetic dataset 3D-Front and real-world dataset Matterport3D. Code and models are available in https://github.com/chtsy/buol.

Figures (10)

• Figure 2: The illustration of our framework. Given a single image, we first predict 2D priors by 2D model, then lift 2D priors to 3D voxels by our occupancy-aware lifting, and finally predict 3D results using the 3D model and obtain panoptic 3D scene reconstruction results in a bottom-up manner.
• Figure 3: Occupancy-aware Lifting. We lift multi-plane occupancy and 2D semantics predicted by the 2D model to 3D features. $\ast$ is Hadamard product.
• Figure 4: Panoptic Reconstruction. The predicted 3D semantics and 3D offsets are first refined by 3D occupancy, and then the reconstructed 3D results are combined with 2D instance centers for 3D instance grouping, and finally, 3D instances and stuff are combined to obtain panoptic 3D scene reconstruction. $\ast$ is Hadamard product.
• Figure 5: Instance Grouping. We convert both 2D instance centers and 3D offsets of each category at multi-plane to group 3D instances.
• Figure 6: Qualitative comparisons against competing methods on 3D-Front. The BUOL and BU denote our Bottom-Up framework w/ and w/o our Occupancy-aware lifting, respectively, and BU-3D denotes the bottom-up framework with instance grouping by 3D centers, and the TD-PD denotes Dahnert et al. dahnert2021panoptic$^*$+PD. And GT is the ground truth.