Multi-View Representation is What You Need for Point-Cloud Pre-Training

TL;DR

This work tackles the 2D-3D domain gap in point-cloud pre-training by learning a 3D feature extractor guided directly by pre-trained 2D networks. The approach projects a 3D feature volume into multiple 2D views to perform cross-modal knowledge transfer with a 2D encoder while simultaneously enforcing geometric consistency across views via a dense correspondence module. The framework, MVNet, combines a 3D SR-UNet-based encoder, a 2D knowledge transfer loss, and a multi-view consistency loss, achieving state-of-the-art or near-state-of-the-art results across semantic segmentation, 3D object detection, shape part segmentation, and shape classification on benchmarks like ScanNet, S3DIS, ShapeNet, and ShapeNetPart. The results demonstrate that leveraging multi-view 2D supervision within a 3D learning framework yields robust, transferable representations for diverse 3D understanding tasks, with practical implications for scalable 3D perception in indoor and potentially outdoor environments.

Abstract

A promising direction for pre-training 3D point clouds is to leverage the massive amount of data in 2D, whereas the domain gap between 2D and 3D creates a fundamental challenge. This paper proposes a novel approach to point-cloud pre-training that learns 3D representations by leveraging pre-trained 2D networks. Different from the popular practice of predicting 2D features first and then obtaining 3D features through dimensionality lifting, our approach directly uses a 3D network for feature extraction. We train the 3D feature extraction network with the help of the novel 2D knowledge transfer loss, which enforces the 2D projections of the 3D feature to be consistent with the output of pre-trained 2D networks. To prevent the feature from discarding 3D signals, we introduce the multi-view consistency loss that additionally encourages the projected 2D feature representations to capture pixel-wise correspondences across different views. Such correspondences induce 3D geometry and effectively retain 3D features in the projected 2D features. Experimental results demonstrate that our pre-trained model can be successfully transferred to various downstream tasks, including 3D shape classification, part segmentation, 3D object detection, and semantic segmentation, achieving state-of-the-art performance.

Figures (6)

• Figure 1: Our model (blue) achieves state-of-the-art performance across a broad range of tasks at both the scene and shape levels. The distance to the origin indicates the task result.
• Figure 2: Approach overview. We used complete and semi-transparent point clouds to represent the input $P$ and the feature volume $F_P$ for better visualization. We encourage readers to frequently reference to this figure while reading Section \ref{['Sec:Method']}.
• Figure 3: 2D knowledge transfer module.$f_p$ is completely frozen during training. For simplicity, we represent the $C$-dimensional feature map $F_i$ using a semi-transparent image.
• Figure 4: Illustration of multi-view consistency module.$\oplus$ denotes side-by-side concatenation.
• Figure 5: Comparison of different 2D pre-trained models.