Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understanding

TL;DR

This work addresses the data scarcity and annotation burden in 3D point cloud understanding by reframing point clouds as specialized images and leveraging large-scale image knowledge through a shared Transformer backbone. The proposed PCExpert architecture freezes the image backbone and introduces a point-specific module that shares the Vision Transformer’s multi-head attention while employing modality-specific FFNs, enabling deep cross-modal knowledge transfer with only a small fraction of trainable parameters. The learning objective combines cross-modal and intra-modal contrastive losses with a novel transformation parameter estimation task, delivering state-of-the-art results on ScanObjectNN in LINEAR and strong performance in few-shot and full-finetuning settings. Overall, PCExpert demonstrates that substantial image-based priors can meaningfully improve 3D understanding, offering a scalable path for multi-modal Transformer-based point cloud learning and dataset-efficient development.

Abstract

Self-supervised representation learning (SSRL) has gained increasing attention in point cloud understanding, in addressing the challenges posed by 3D data scarcity and high annotation costs. This paper presents PCExpert, a novel SSRL approach that reinterprets point clouds as "specialized images". This conceptual shift allows PCExpert to leverage knowledge derived from large-scale image modality in a more direct and deeper manner, via extensively sharing the parameters with a pre-trained image encoder in a multi-way Transformer architecture. The parameter sharing strategy, combined with a novel pretext task for pre-training, i.e., transformation estimation, empowers PCExpert to outperform the state of the arts in a variety of tasks, with a remarkable reduction in the number of trainable parameters. Notably, PCExpert's performance under LINEAR fine-tuning (e.g., yielding a 90.02% overall accuracy on ScanObjectNN) has already approached the results obtained with FULL model fine-tuning (92.66%), demonstrating its effective and robust representation capability.

Figures (3)

• Figure 1: The pipeline of PCExpert. Left: The input representations consist of sequences of embeddings, which are the summation of the patch/CLS tokens, the type embeddings and the position embeddings for the respective point and image data. Middle: The point and image input representations are then fed into a series of transformer blocks. In each block, the representations are first processed by a shared Vision Transformer (ViT) Multi-head Self-Attention (MSA) module, and then processed by separate Feed Forward Networks (FFNs), according to their modality. Right: During the pre-training process, the parameters in ViT are kept frozen, while only the parameters related to point processing and projection heads are optimized, via three objectives: cross-modal contrastive ($\mathcal{L}_{cm}$), intra-modal contrastive ($\mathcal{L}_{im}$) and rotation angle regression ($\mathcal{L}_{reg}$).
• Figure 2: Training samples used in point-image contrastive learning. Left: Point cloud samples. Middle: Images rendered from 3D CAD meshes. Right: Images rendered directly from the original point clouds, with the shape and details well preserved.
• Figure 3: left: Gradient calculation is based on $\mathcal{L}_{cm}$ and $\mathcal{L}_{reg}$, excluding $\mathcal{L}_{im}$. Optimizing for $\mathcal{L}_{reg}$ (the red curve) concurrently results in a reduction of $\mathcal{L}_{im}$ (green). right: Gradient calculation is based on $\mathcal{L}_{cm}$ and $\mathcal{L}_{im}$, excluding $\mathcal{L}_{reg}$. Optimizing for $\mathcal{L}_{im}$ has no effect on $\mathcal{L}_{reg}$.