Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition

Dan Song, Xinwei Fu, Ning Liu, Weizhi Nie, Wenhui Li, Lanjun Wang, You Yang, Anan Liu

TL;DR

MV-CLIP addresses zero-shot 3D shape recognition by bridging the 2D-3D gap through a multi-view CLIP framework. It introduces two key components: entropy-based view selection to filter semantically clear views and a hierarchical prompts scheme that first votes candidates with hand-crafted prompts and then refines with LLM-generated prompts, all without 3D training. The approach achieves strong zero-shot accuracies on ModelNet40, ModelNet10, and ShapeNet Core55, and ablations demonstrate complementary gains from view selection and hierarchical prompts. The work demonstrates a practical path to open-world 3D understanding by effectively repurposing large-scale vision-language models for 3D via prompt engineering and view-level uncertainty reasoning, with potential for extension to newer backbones and prompts.

Abstract

Large-scale pre-trained models have demonstrated impressive performance in vision and language tasks within open-world scenarios. Due to the lack of comparable pre-trained models for 3D shapes, recent methods utilize language-image pre-training to realize zero-shot 3D shape recognition. However, due to the modality gap, pretrained language-image models are not confident enough in the generalization to 3D shape recognition. Consequently, this paper aims to improve the confidence with view selection and hierarchical prompts. Leveraging the CLIP model as an example, we employ view selection on the vision side by identifying views with high prediction confidence from multiple rendered views of a 3D shape. On the textual side, the strategy of hierarchical prompts is proposed for the first time. The first layer prompts several classification candidates with traditional class-level descriptions, while the second layer refines the prediction based on function-level descriptions or further distinctions between the candidates. Remarkably, without the need for additional training, our proposed method achieves impressive zero-shot 3D classification accuracies of 84.44%, 91.51%, and 66.17% on ModelNet40, ModelNet10, and ShapeNet Core55, respectively. Furthermore, we will make the code publicly available to facilitate reproducibility and further research in this area.

MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition

TL;DR

MV-CLIP addresses zero-shot 3D shape recognition by bridging the 2D-3D gap through a multi-view CLIP framework. It introduces two key components: entropy-based view selection to filter semantically clear views and a hierarchical prompts scheme that first votes candidates with hand-crafted prompts and then refines with LLM-generated prompts, all without 3D training. The approach achieves strong zero-shot accuracies on ModelNet40, ModelNet10, and ShapeNet Core55, and ablations demonstrate complementary gains from view selection and hierarchical prompts. The work demonstrates a practical path to open-world 3D understanding by effectively repurposing large-scale vision-language models for 3D via prompt engineering and view-level uncertainty reasoning, with potential for extension to newer backbones and prompts.

Abstract

Large-scale pre-trained models have demonstrated impressive performance in vision and language tasks within open-world scenarios. Due to the lack of comparable pre-trained models for 3D shapes, recent methods utilize language-image pre-training to realize zero-shot 3D shape recognition. However, due to the modality gap, pretrained language-image models are not confident enough in the generalization to 3D shape recognition. Consequently, this paper aims to improve the confidence with view selection and hierarchical prompts. Leveraging the CLIP model as an example, we employ view selection on the vision side by identifying views with high prediction confidence from multiple rendered views of a 3D shape. On the textual side, the strategy of hierarchical prompts is proposed for the first time. The first layer prompts several classification candidates with traditional class-level descriptions, while the second layer refines the prediction based on function-level descriptions or further distinctions between the candidates. Remarkably, without the need for additional training, our proposed method achieves impressive zero-shot 3D classification accuracies of 84.44%, 91.51%, and 66.17% on ModelNet40, ModelNet10, and ShapeNet Core55, respectively. Furthermore, we will make the code publicly available to facilitate reproducibility and further research in this area.
Paper Structure (31 sections, 8 equations, 11 figures, 9 tables)

This paper contains 31 sections, 8 equations, 11 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Zero-shot Learning in 3D Shape Recognition
  4. Multi-View in 3D Shape Recognition
  5. Prompt Learning in Vision
  6. Methodology
  7. View Rendering and Feature Extraction
  8. View Selection
  9. Hierarchical Prompts
  10. 3D-Specific Prompts Powered by LLMs
  11. Hierarchical Prompts Matching
  12. Experiments
  13. Dataset
  14. Experimental Setting and Details
  15. Zero-shot 3D recognition
  16. ...and 16 more sections

Figures (11)

  • Figure 1: Improve CLIP's confidence at zero-shot 3D shape recognition in the visual aspect: select images with clear semantics.
  • Figure 2: Improve CLIP's confidence at zero-shot 3D shape recognition in the textual prompt: refine the prediction with hierarchical prompts. Left: An example of bathtub that is mis-classified into sink. Right: Statistical zero-shot top-k accuracy on three popular datasets, which is obtained by the CLIP model under the settings of 12 pre-defined online-rendered views and hand-crafted prompts of first layer.
  • Figure 3: Overview of the proposed MV-CLIP for zero-shot 3D shape recognition. Firstly, multiple view images are obtained via a render $\mathbf{R}$ and the corresponding visual features are extracted via the visual encoder of CLIP 13_clip. Secondly, visual features are matched with textual features encoded by CLIP with hand-crafted prompts, and we select representative views according the prediction confidence. By aggregating the representative predictions, several candidates with the top classification probability are kept for the second matching. Finally, by matching the prompts powered by LLMs for these candidates, the prediction result is refined.
  • Figure 4: The decisions of the selected 4 views of $40\times4$ randomly chosen samples on ModelNet40.
  • Figure 5: View selection
  • ...and 6 more figures