Transformed Multi-view 3D Shape Features with Contrastive Learning

TL;DR

This work addresses 3D shape understanding from multi-view images by integrating Vision Transformer backbones with a range of supervised and self-supervised contrastive losses in a two-stage pipeline (multi-view rendering followed by downstream tasks). It conducts a comprehensive evaluation across four ViT backbones and five contrastive losses, achieving notable results such as $90.6\%$ accuracy on ModelNet10 and $95.5\%$ mAP for retrieval on ModelNet10, surpassing several CNN-based and prior contrastive approaches. The findings show ViTs can capture global shape semantics effectively, while contrastive objectives refine local discriminative features, yielding data-efficient representations and strong retrieval performance. The study also identifies limitations, such as persistent confusion among semantically related classes, and proposes future directions like multi-scale ViTs and relational contrastive mechanisms to further improve 3D representation learning.

Abstract

This paper addresses the challenges in representation learning of 3D shape features by investigating state-of-the-art backbones paired with both contrastive supervised and self-supervised learning objectives. Computer vision methods struggle with recognizing 3D objects from 2D images, often requiring extensive labeled data and relying on Convolutional Neural Networks (CNNs) that may overlook crucial shape relationships. Our work demonstrates that Vision Transformers (ViTs) based architectures, when paired with modern contrastive objectives, achieve promising results in multi-view 3D analysis on our downstream tasks, unifying contrastive and 3D shape understanding pipelines. For example, supervised contrastive losses reached about 90.6% accuracy on ModelNet10. The use of ViTs and contrastive learning, leveraging ViTs' ability to understand overall shapes and contrastive learning's effectiveness, overcomes the need for extensive labeled data and the limitations of CNNs in capturing crucial shape relationships. The success stems from capturing global shape semantics via ViTs and refining local discriminative features through contrastive optimization. Importantly, our approach is empirical, as it is grounded on extensive experimental evaluation to validate the effectiveness of combining ViTs with contrastive objectives for 3D representation learning.

Figures (4)

• Figure 1: Overview of the experimental pipeline, consisting of two stages. Stage 01 performs multi-view rendering from 3D objects (e.g., ModelNet10/40), followed by contrastive learning using different loss functions (e.g., SupCon, InfoNCE, SimCLR, SINCERE, and $\epsilon$-SupInfoNCE) and backbones. Embeddings can be frozen or fine-tuned. Stage 02 evaluates the learned representations through downstream tasks such as 3D shape classification and retrieval. The entire pipeline is designed to answer our three main research questions.
• Figure 2: Multi-view rendering acquisition from 3D objects relies on our camera setup. Some representative examples from the airplane, guitar, and table classes to illustrate the diversity of views generated.
• Figure 3: Qualitative comparison of Top-10 retrieved 3D objects using our best contrastive models on the ModelNet10 and ModelNet40 datasets. Each row shows retrieval results for a given model, top to bottom (SINCERE with ViT-B/16, SimCLR with DINO-B/16, $\epsilon$-SupInfoNCE with DINO-S/16, and SimCLR with DINO-B/16). For each query (leftmost column), the ten most similar objects are shown from left to right. Red boxes indicate incorrect class retrievals. SINCERE and $\epsilon$-SupInfoNCE yield more consistent and semantically accurate retrievals, highlighting their stronger representation learning capacity.
• Figure 4: t-SNE visualization of the representation spaces obtained with the best contrastive learning settings after linear evaluation. Each row shows a number in the embedding space, which means: SINCERE with ViT-B/16, SimCLR with DINO-B/16, $\epsilon$-SupInfoNCE with DINO-S/16, and SimCLR with DINO-B/16. Red circles highlight areas with misclustered class overlap, and the zoomed-in views below offer a detailed look into such regions.