Duoduo CLIP: Efficient 3D Understanding with Multi-View Images

TL;DR

This paper addresses 3D shape understanding by replacing point-cloud encoders with a multi-view image encoder that leverages pretrained 2D CLIP priors. It introduces Duoduo CLIP, which uses a multi-view shape encoder initialized from CLIP's image encoder, incorporates cross-view attention across a variable number of views $M$, and optimizes a contrastive loss among shape, image, and text modalities, averaging per-view embeddings into a single shape representation. The approach achieves state-of-the-art performance on real and synthetic datasets (e.g., Objaverse-LVIS, MVPNet, ScanObjectNN, MVImgNet) with significantly reduced compute and memory relative to point-cloud methods. These results indicate that multi-view image representations can yield better 3D-text alignment and practical scalability, with future work exploring depth/normal inputs and other SSL frameworks.

Abstract

We introduce Duoduo CLIP, a model for 3D representation learning that learns shape encodings from multi-view images instead of point clouds. The choice of multi-view images allows us to leverage 2D priors from off-the-shelf CLIP models to facilitate fine-tuning with 3D data. Our approach not only shows better generalization compared to existing point cloud methods, but also reduces GPU requirements and training time. In addition, the model is modified with cross-view attention to leverage information across multiple frames of the object which further boosts performance. Notably, our model is permutation invariant to the order of multi-view images while being pose-free. Compared to the current SOTA point cloud method that requires 480 A100 hours to train 1 billion model parameters we only require 57 A5000 hours and 87 million parameters. Multi-view images also provide more flexibility including being able to encode objects with a variable number of images, and performance scales when more views are used. In contrast, point cloud based methods require an entire scan or model of the object. We showcase this flexibility with benchmarks from images of real-world objects. Our model also achieves better performance in more fine-grained text to shape retrieval, demonstrating better text-and-shape alignment than point cloud based models.

Figures (12)

• Figure 1: Duoduo CLIP is a model for 3D shape understanding that represents the shape with a multi-view images encoder instead of a point-cloud (PC) encoder. This makes our model more accurate and efficient than PC methods (OpenShape liu2024openshape, Uni3D zhou2023uni3d).
• Figure 2: We blend concepts from two distinct objects by identifying the Objaverse shape with the embedding that maximizes similarity to both objects simultaneously. This shows that our model has learned a latent space capable of combining geometric features from the mixed objects or capturing their semantic attributes. For more details see \ref{['appendix:concept_mixing']}.
• Figure 3: Our multi-view encoder takes a variable number $[1, M]$ of images as input and outputs a single-shape embedding. The first few layers of the ViT model are frozen (blue) and attention operates on the individual frames in parallel. For the latter layers, the attention layers are trainable (orange) and modified to attend over all $M$ views. The embeddings for each frame are averaged to get the final embedding. Note that only major components of the model are depicted.
• Figure 4: Attention maps are extracted from the first trainable MVA layer, capturing token-wise attention across all M views. A specific token (e.g., the plate) is queried to retrieve the corresponding row from the attention map, and its magnitudes are mapped onto the original images for visualization. The model has learned geometric correspondences of 3D shapes across different multi-views.
• Figure 5: Left. We compare the Top 1 accuracies on Objaverse-LVIS for our model (Duoduo CLIP) and two pretrained CLIP models (zero-shot) for different number of frames. B-32 is the model we initialize from and train with and BigG-14 is the model used by OpenShape. The dashed lines show the difference between using 1 and 12 frames. Right. We show the Top1, Top3 and Top5 accuracies for our model for 1 to 12 frames. Gray dashed lines show the inference time(s) for 1 iteration and batch size 200 at different number of frames.