BelHouse3D: A Benchmark Dataset for Assessing Occlusion Robustness in 3D Point Cloud Semantic Segmentation

TL;DR

The BelHouse3D dataset is introduced, a new synthetic point cloud dataset designed for 3D indoor scene semantic segmentation and its OOD setting is evaluated, providing valuable insights for the development of more generalizable models.

Abstract

Large-scale 2D datasets have been instrumental in advancing machine learning; however, progress in 3D vision tasks has been relatively slow. This disparity is largely due to the limited availability of 3D benchmarking datasets. In particular, creating real-world point cloud datasets for indoor scene semantic segmentation presents considerable challenges, including data collection within confined spaces and the costly, often inaccurate process of per-point labeling to generate ground truths. While synthetic datasets address some of these challenges, they often fail to replicate real-world conditions, particularly the occlusions that occur in point clouds collected from real environments. Existing 3D benchmarking datasets typically evaluate deep learning models under the assumption that training and test data are independently and identically distributed (IID), which affects the models' usability for real-world point cloud segmentation. To address these challenges, we introduce the BelHouse3D dataset, a new synthetic point cloud dataset designed for 3D indoor scene semantic segmentation. This dataset is constructed using real-world references from 32 houses in Belgium, ensuring that the synthetic data closely aligns with real-world conditions. Additionally, we include a test set with data occlusion to simulate out-of-distribution (OOD) scenarios, reflecting the occlusions commonly encountered in real-world point clouds. We evaluate popular point-based semantic segmentation methods using our OOD setting and present a benchmark. We believe that BelHouse3D and its OOD setting will advance research in 3D point cloud semantic segmentation for indoor scenes, providing valuable insights for the development of more generalizable models.

Figures (6)

• Figure 1: BelHouse3D is a synthetic 3D point cloud dataset specifically designed for indoor scene semantic segmentation. It provides clean data and precise annotations for model training and in-distribution (IID) testing. Additionally, it includes a test set designed to simulate real-world occlusion, serving as an out-of-distribution (OOD) benchmark. The left side of the figure displays a scene from the training dataset, representing House 6, while the right side illustrates the OOD test data with occlusion from House 30.
• Figure 2: Bar graph illustrating the data distribution for the BelHouse3D dataset, with building structures comprising approximately $\approx64\%$, household objects $\approx22\%$, and clutter $\approx14\%$ of the total annotations. This distribution emphasizes the dataset's focus on capturing both the structural and functional elements of indoor scenes while addressing the challenge of segmenting clutter, a significant factor in real-world applications. Additionally, bubble sizes represent the number of points in each class, with the values (in millions) for the six major classes displayed.
• Figure 3: Workflow outlining the stages in creating the BelHouse3D dataset. The process starts with capturing image frames and proceeds through the following stages: point cloud reconstruction, creation of Blender files based on real data references, generation of object files, and final sampling to create the point clouds.
• Figure 4: Synthetic point clouds generated for House 28 from the BelHouse3D dataset, comparing the IID test data (top) and the OOD test data with occlusion (bottom). The bedroom scene is depicted with and without furniture to highlight the impact of occlusion on building structures.
• Figure 5: Qualitative results of fully supervised 3D segmentation methods evaluated on House 30 from the BelHouse3D dataset. The performance decline is evident across all methods when transitioning from the IID test set to the OOD test set with occlusions.