CLIP-based Point Cloud Classification via Point Cloud to Image Translation

TL;DR

The paper tackles the challenge of classifying 3D point clouds by extending CLIP-based vision-language models to 3D data. It introduces PPCITNet, a Pretrained Point Cloud to Image Translation Network, which learns to transform point cloud depth maps into generalized colored images with salient cues by pretraining on binary mask–RGB pairs, thereby aligning depth features with CLIP visuals. A novel viewpoint adapter then fuses per-view features with global multi-view information, enabling effective few-shot learning while keeping the base CLIP model fixed. Empirical results on ModelNet10, ModelNet40, and ScanObjectNN show state-of-the-art performance in zero-shot and few-shot settings, with strong robustness on real-world data, demonstrating the practical value of bridging depth maps and natural images for 3D recognition.

Abstract

Point cloud understanding is an inherently challenging problem because of the sparse and unordered structure of the point cloud in the 3D space. Recently, Contrastive Vision-Language Pre-training (CLIP) based point cloud classification model i.e. PointCLIP has added a new direction in the point cloud classification research domain. In this method, at first multi-view depth maps are extracted from the point cloud and passed through the CLIP visual encoder. To transfer the 3D knowledge to the network, a small network called an adapter is fine-tuned on top of the CLIP visual encoder. PointCLIP has two limitations. Firstly, the point cloud depth maps lack image information which is essential for tasks like classification and recognition. Secondly, the adapter only relies on the global representation of the multi-view features. Motivated by this observation, we propose a Pretrained Point Cloud to Image Translation Network (PPCITNet) that produces generalized colored images along with additional salient visual cues to the point cloud depth maps so that it can achieve promising performance on point cloud classification and understanding. In addition, we propose a novel viewpoint adapter that combines the view feature processed by each viewpoint as well as the global intertwined knowledge that exists across the multi-view features. The experimental results demonstrate the superior performance of the proposed model over existing state-of-the-art CLIP-based models on ModelNet10, ModelNet40, and ScanobjectNN datasets.

Figures (5)

• Figure 1: Example of different image representations: (a) natural RGB images; (b) rendered RGB images; (c) point cloud depth maps; (d) 3D depth maps; (e) processed binary mask images.
• Figure 2: For a binary mask image, we multiply the binary image with a noise image to the make binary image sparse. The noise image is composed of $50\%$ white pixel and $50\%$ of black pixel sampled randomly.
• Figure 3: The training of our approach is composed of two stages. In the first stage, we pre-train our PCITNet using the processed binary mask and RGB pairs. In the second stage, we perform a few shot learning on a viewpoint adapter utilizing PPCITNet and pre-trained CLIP.
• Figure 4: Input Visualization. Our PPCITNet produces generalized colored images along with additional salient visual cues. The salient visual cues refer to additional color concentration to prominent or distinctive parts of the image.
• Figure 5: Few-shot performance comparison under 1, 2, 4, 8,10,12,14, and 16-shot settings.