To Supervise or Not to Supervise: Understanding and Addressing the Key Challenges of Point Cloud Transfer Learning

TL;DR

This work quantitatively compares supervised and contrastive pre-training for 3D point-cloud transfer learning across diverse architectures and datasets, revealing that early 3D layers can be class-discriminative and that supervised pre-training often excels in linear probing while contrastive pre-training can outperform supervised with full fine-tuning. It introduces a simple layer-wise geometric regularization based on predicting point normals to enhance the adaptability of early layers, demonstrated to improve transfer across ModelNet40, ScanObjectNN, and S3DIS. The authors provide a consistent evaluation framework, thorough feature analyses (including layer-gradient norms and t-SNE visualizations), and show that the regularization improves performance and increases early-layer gradient norms, suggesting better downstream adaptability. These findings offer practical guidance for designing robust 3D pre-training pipelines and point to future work in architecture design and multi-scale geometric representations for transfer learning.

Abstract

Transfer learning has long been a key factor in the advancement of many fields including 2D image analysis. Unfortunately, its applicability in 3D data processing has been relatively limited. While several approaches for point cloud transfer learning have been proposed in recent literature, with contrastive learning gaining particular prominence, most existing methods in this domain have only been studied and evaluated in limited scenarios. Most importantly, there is currently a lack of principled understanding of both when and why point cloud transfer learning methods are applicable. Remarkably, even the applicability of standard supervised pre-training is poorly understood. In this work, we conduct the first in-depth quantitative and qualitative investigation of supervised and contrastive pre-training strategies and their utility in downstream 3D tasks. We demonstrate that layer-wise analysis of learned features provides significant insight into the downstream utility of trained networks. Informed by this analysis, we propose a simple geometric regularization strategy, which improves the transferability of supervised pre-training. Our work thus sheds light onto both the specific challenges of point cloud transfer learning, as well as strategies to overcome them.

Figures (10)

• Figure 1: Analyzing and improving point cloud transfer learning. In this work, we perform the first in-depth study of the different components of network pre-training that influence the outcome of point cloud transfer learning (components in solid orange). This includes the source data domain in relation to the downstream target data, the choice of architecture, the importance of early vs later layers, and the pre-training design choices. We also show that improvements to the pipeline can be achieved through regularization of the early layers, by promoting the prediction of geometric properties.
• Figure 2: Evaluation on (a) ModelNet40 and (b) ScanObjectNN classification tasks of different pre-trained models, using linear probing (LP -- solid bars) and fine-tuning (FT -- dashed bars) settings. Random Init is a randomly initialized model. Transfer learning performance depends on pre-training scheme, architecture and evaluation protocol.
• Figure 3: t-SNE plots of the first-layer feature activation for different architectures and pre-training schemes. We use the ModelNet10 evaluation set, which is a subset of ModelNet40 containing 10 classes, each represented by a different color. Clusters are formed even in the feature space of first layers, which implies their discriminative capability. Visualization on additional architectures can be found in the supplementary.
• Figure 4: Layer-wise gradient norms of pre-trained models on downstream ModelNet40 and ScanObjectNN datasets using DGCNN and PointNet. Supervised pre-training shows low gradient norms, especially in early layers. Further analysis across architectures is provided in the supplementary.
• Figure 5: Layer-wise gradient norms of supervised pre-trained models with and without regularization on downstream ModelNet40 and ScanObjectNN datasets using DGCNN. Regularization increases the gradient norm values, especially in early layers.