Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training

TL;DR

MixCon3D addresses the need for holistic 3D open-world understanding by integrating multi-view 2D images with 3D point clouds and aligning the joint 3D representation to text within a contrastive learning framework. It introduces a simple yet effective 3D sculptor that fuses multi-view image features with point-cloud features to form $z_i^{3D}$, and adds a dedicated 3D–text contrastive loss alongside established image–text and 3D–text alignments. The authors also provide a thorough exploration of training recipes (batch size, LR schedules, EMA, separate temperatures) and demonstrate significant gains on Objaverse-LVIS (+5.7% Top1) and ScanObjectNN (+6.4%) across encoders, plus strong cross-modal capabilities such as text-to-3D retrieval and point-cloud captioning. Overall, MixCon3D advances open-world 3D understanding by leveraging cross-modal complementarities and structured 3D representations, with practical implications for retrieval and captioning tasks in multimodal AI systems.

Abstract

Contrastive learning has emerged as a promising paradigm for 3D open-world understanding, i.e., aligning point cloud representation to image and text embedding space individually. In this paper, we introduce MixCon3D, a simple yet effective method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training. In contrast to point cloud only, we develop the 3D object-level representation from complementary perspectives, e.g., multi-view rendered images with the point cloud. Then, MixCon3D performs language-3D contrastive learning, comprehensively depicting real-world 3D objects and bolstering text alignment. Additionally, we pioneer the first thorough investigation of various training recipes for the 3D contrastive learning paradigm, building a solid baseline with improved performance. Extensive experiments conducted on three representative benchmarks reveal that our method significantly improves over the baseline, surpassing the previous state-of-the-art performance on the challenging 1,156-category Objaverse-LVIS dataset by 5.7%. The versatility of MixCon3D is showcased in applications such as text-to-3D retrieval and point cloud captioning, further evidencing its efficacy in diverse scenarios. The code is available at https://github.com/UCSC-VLAA/MixCon3D.

Figures (6)

• Figure 1: Comparison of zero-shot point cloud recognition between the OpenShape (blue) and our MixCon3D (red) under different pre-training datasets (ShapeNet, Ensemble (No LVIS) and Ensemble). Our model obtains consistent improvements on different training datasets on various downstream benchmarks.
• Figure 2: Summary of our MixCon3D framework. MixCon3D first extracts the representation of input triplets (images, text, point cloud) from a pre-trained vision-language model (e.g., CLIP) and a 3D encoder (e.g., Point-BERT) with corresponding projection heads. Then, the image and point cloud features go through a 3D sculptor to obtain the 3D object-level features, serving as complementary representations. The contrastive losses are applied to align features among three modalities (image-text-3D) and 3D representation to text.
• Figure 3: Analysis of the number in the multi-view mechanism. We report the Top 1 Accuracy results from 1 to 12 views.
• Figure 4: Text to 3D object retrieval comparisons. The input text and the first three retrieved 3D objects are listed.
• Figure 5: Point cloud captioning comparisons. In each row, we list the input point cloud, corresponding images, and generated captions.