CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

TL;DR

CG3D addresses the lack of language-grounded 3D foundation models by embedding a trainable 3D point-cloud encoder into CLIP and aligning it with CLIP's image and text representations through cross-modal contrastive losses. It circumvents catastrophic forgetting during adaptation by freezing CLIP components and using deep visual prompt tuning to shift input distributions toward CLIP's training data. The framework demonstrates strong zero-shot 3D recognition, language-based scene querying, and cross-modal 3D retrieval, while providing competitive starting weights for downstream supervised tasks. Overall, CG3D advances open-world 3D understanding and practical 3D retrieval by grounding 3D shapes in natural language and visual semantics, with promising potential for robotics and AR applications.

Abstract

Vision-Language models like CLIP have been widely adopted for various tasks due to their impressive zero-shot capabilities. However, CLIP is not suitable for extracting 3D geometric features as it was trained on only images and text by natural language supervision. We work on addressing this limitation and propose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder is learned to exhibit zero-shot capabilities. CG3D is trained using triplets of pointclouds, corresponding rendered 2D images, and texts using natural language supervision. To align the features in a multimodal embedding space, we utilize contrastive loss on 3D features obtained from the 3D encoder, as well as visual and text features extracted from CLIP. We note that the natural images used to train CLIP and the rendered 2D images in CG3D have a distribution shift. Attempting to train the visual and text encoder to account for this shift results in catastrophic forgetting and a notable decrease in performance. To solve this, we employ prompt tuning and introduce trainable parameters in the input space to shift CLIP towards the 3D pre-training dataset utilized in CG3D. We extensively test our pre-trained CG3D framework and demonstrate its impressive capabilities in zero-shot, open scene understanding, and retrieval tasks. Further, it also serves as strong starting weights for fine-tuning in downstream 3D recognition tasks.

Figures (9)

• Figure 1: Overview of our proposed framework CLIP goes 3D (CG3D). We introduce a 3D Encoder in the CLIP framework and pre-train it using natural language supervision while also leveraging CLIP's pre-trained visual encoder. CG3D solves various practical tasks like zero-shot 3D recognition, 3D point cloud retrieval, scene querying with natural language, Moreover, it can serve as a strong initial weight for standard fine-tuning tasks.
• Figure 2: Overview of the proposed learning strategy in CG3D. Note that only the 3D Encoder and learnable visual prompts are trained while everything else is frozen.
• Figure 3: Example of scene querying with text for a random indoor scene from S3DIS 2017arXiv170201105A dataset.
• Figure 4: Retrieving point clouds from a 3D database (ModelNet40) using random image and text queries.
• Figure 5: Experiments on data scarce setups on ModelNet40 with PointTransformer and PointMLP backbones.