PointCLIP: Point Cloud Understanding by CLIP

TL;DR

PointCLIP transfers CLIP's 2D image–text priors to 3D point clouds by projecting points into multi-view depth maps and using a zero-shot CLIP classifier. An inter-view adapter enables efficient few-shot adaptation and cross-view fusion, while a multi-knowledge ensembling strategy leverages complementary information from classical 3D networks. Experiments on ModelNet10/40 and ScanObjectNN show competitive zero-shot performance and strong few-shot gains, with ensembling yielding state-of-the-art results in several cases. The approach offers a data-efficient, low-cost path to 3D understanding by reusing large-scale 2D vision–language knowledge and acting as a plug-in module for existing 3D architectures.

Abstract

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.

Figures (6)

• Figure 1: A Comparison of Training-testing schemes between PointCLIP and PointNet++. Different from classical 3D networks, our proposed PointCLIP is pre-trained by 2D image-text pairs, but conducts zero-shot classification on 3D datasets, which achieves cross-modality knowledge transfer.
• Figure 2: The Pipeline of PointCLIP. To bridge the modal gap, PointCLIP projects the point cloud onto multi-view depth maps, and conducts 3D recognition via CLIP pre-trained in 2D. The switch provides alternatives for direct zero-shot classification and few-shot classification with inter-view adapter, respectively, in solid and dotted lines.
• Figure 3: Detailed structure of the proposed Inter-view Adapter. Given multi-view features of a point cloud, the adapter extracts its global representation and generates view-wise adapted features. Via a residual connection, the newly-learned 3D knowledge is fused into the pre-trained CLIP.
• Figure 4: PointCLIP could provide complimentary 2D knowledge to classical 3D networks and serve as a plug-and-play enhancement module.
• Figure 5: Few-shot performance comparison between PointCLIP and other classical 3D networks, including the state-of-the-art CurveNet, on ModelNet10, ModelNet40 and ScanObjectNN. Our PointCLIP shows consistent superiority to other models under 1, 2, 4, 8 and 16-shot settings.