Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding

TL;DR

This work tackles the data scarcity challenge in self-supervised 3D scene understanding by proposing Generalizable Representation Learning (GRL), which leverages a generative Bayesian network to synthesize diverse, real-world–patterned scenes and a joint learning objective that enforces both coarse-to-fine object- and point-level contrastive cues plus occlusion-aware reconstruction. The synthetic pretraining yields geometry-aware representations that transfer effectively to downstream tasks, notably 3D object detection and semantic segmentation, across multiple benchmarks and data-efficient settings. The approach achieves consistent gains over state-of-the-art pretraining methods, with ablations confirming the contributions of synthetic data generation, occlusion handling, and the multi-task objective. Overall, GRL demonstrates strong generalization capabilities for data-efficient 3D scene understanding with practical impact for robotics, autonomous systems, and AR/VR applications.

Abstract

The field of self-supervised 3D representation learning has emerged as a promising solution to alleviate the challenge presented by the scarcity of extensive, well-annotated datasets. However, it continues to be hindered by the lack of diverse, large-scale, real-world 3D scene datasets for source data. To address this shortfall, we propose Generalizable Representation Learning (GRL), where we devise a generative Bayesian network to produce diverse synthetic scenes with real-world patterns, and conduct pre-training with a joint objective. By jointly learning a coarse-to-fine contrastive learning task and an occlusion-aware reconstruction task, the model is primed with transferable, geometry-informed representations. Post pre-training on synthetic data, the acquired knowledge of the model can be seamlessly transferred to two principal downstream tasks associated with 3D scene understanding, namely 3D object detection and 3D semantic segmentation, using real-world benchmark datasets. A thorough series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.

Figures (5)

• Figure 1: Comparison of representation learning methodologies. We compare our overall framework with CSC partition and RandomRooms randomrooms.
• Figure 2: Overall structure of pre-training framework. Our GRL framework devises a generative Bayesian network as the scene generator, whose generated synthetic scenes are augmented with occlusion. Subsequently, we jointly learn a coarse-to-fine contrastive learning task and an occlusion-aware reconstruction task. Note that we show full points of $\bm{z}_A$ and $\bm{z}_B$ for better visualization.
• Figure 3: Our scene generator is modeled as a generative Bayesian network. $N_{ps}$ is the number of objects per scene.
• Figure 4: Illustration of our relaxed point pair matching scheme.
• Figure 5: Qualitative results of 3D object detection (first row) and 3D semantic segmentation (second row) when fine-tuned with 10% of training data. The colors of bounding boxes and segmented points represent the semantic labels. Note that the two tasks are fine-tuned individually, initialized with the same pre-trained PointNet++ parameters.