Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining

TL;DR

3D recognition suffers from limited labeled data; this work presents a cross-modal pretraining approach that transfers knowledge from 2D pretrained networks to 3D models using Contrastive Pixel-to-Point Knowledge Transfer (PPKT). By aligning 2D pixel features with 3D point features through a differentiable back-projection and a learnable UpSampling Projection Layer, trained with a local pixel-point contrastive loss, the method provides effective 3D initial weights from unlabeled RGB-D data. Empirical results across S3DIS, SUN RGB-D, and ScanNet demonstrate consistent gains over training from scratch and competitive baselines, with larger 3D backbones and data-limited fine-tuning scenarios benefiting most. This approach complements prior 3D self-supervised methods like PointContrast and highlights the value of leveraging abundant 2D supervision for 3D pretraining and downstream scene understanding tasks.

Abstract

Most 3D neural networks are trained from scratch owing to the lack of large-scale labeled 3D datasets. In this paper, we present a novel 3D pretraining method by leveraging 2D networks learned from rich 2D datasets. We propose the contrastive pixel-to-point knowledge transfer to effectively utilize the 2D information by mapping the pixel-level and point-level features into the same embedding space. Due to the heterogeneous nature between 2D and 3D networks, we introduce the back-projection function to align the features between 2D and 3D to make the transfer possible. Additionally, we devise an upsampling feature projection layer to increase the spatial resolution of high-level 2D feature maps, which enables learning fine-grained 3D representations. With a pretrained 2D network, the proposed pretraining process requires no additional 2D or 3D labeled data, further alleviating the expensive 3D data annotation cost. To the best of our knowledge, we are the first to exploit existing 2D trained weights to pretrain 3D deep neural networks. Our intensive experiments show that the 3D models pretrained with 2D knowledge boost the performances of 3D networks across various real-world 3D downstream tasks.

Figures (5)

• Figure 1: Learning from 2D as 3D pretraining. Due to the limited size of 3D labeled data, 3D pretraining with large unlabeled data is critical. (a) Previous works xie2020pointcontrast apply self-supervised learning on 3D as pretraining. (b) We propose pretraining 3D networks by leveraging the existing 2D network weights with our contrastive pixel-to-point knowledge transfer. In this way, we can provide the knowledge learned from rich 2D image datasets for the 3D networks as initial model weights.
• Figure 2: Contrastive pixel-to-point knowledge transfer (PPKT). PPKT transfers the 2D pretrained network knowledge into 3D from pixels to points. A back-projection is used to align corresponding pixel-level features and point-level features. To restore the granularity of low-resolution 2D feature maps, we propose the learnable upsampling feature projection layer (UPL). The details are described in Section \ref{['sec:ppkt']}.
• Figure 3: T-SNE plot of global features versus pixel-level features. For the images in ScanNet scene0000_00, the global features of images extracted from ImageNet pretrained ResNet are less discriminative in feature space than pixel-level features. Therefore, we believe that pixel-level features are more favorable for knowledge transfer and can preserve fine-grained pixel-level information.
• Figure 4: Limited labeled data fine-tuning on ScanNet semantic segmentation. We sub-sample the labeled scenes in the ScanNet dataset into 50%, 30%, and 15%. With less labeled data available, the gap between our PPKT pretraining and training from scratch becomes larger.
• Figure 5: T-SNE visualization of point-level features in ScanNet. We visualize the last-layer point features of a 3D scene using different 3D network weights. The feature points are colored with segmentation class labels. Without any 3D labeled data, our method shows the high-level semantic understanding ability by learning from 2D rich information.