Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes

Alexandros Delitzas, Maria Parelli, Nikolas Hars, Georgios Vlassis, Sotirios Anagnostidis, Gregor Bachmann, Thomas Hofmann

TL;DR

The paper tackles how to transfer large-scale 2D vision-language knowledge to 3D scene reasoning. It introduces Multi-CLIP, a pre-training approach that aligns 3D scene representations with CLIP's text and multi-view image embeddings via a contrastive objective, aided by an auxiliary detection loss. Through pre-training a 3D scene encoder and transferring it to downstream 3D-VQA and 3D-SQA models, the method achieves state-of-the-art results on ScanQA and SQA3D and demonstrates a semantically structured 3D feature space. This work highlights the practical potential of cross-modal 2D-3D knowledge transfer for embodied 3D reasoning tasks.

Abstract

Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. However, it still remains understudied whether 2D distilled knowledge can provide useful representations for downstream 3D vision-language tasks such as 3D question answering. In this paper, we propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations. We leverage the representational power of the CLIP model by maximizing the agreement between the encoded 3D scene features and the corresponding 2D multi-view image and text embeddings in the CLIP space via a contrastive objective. To validate our approach, we consider the challenging downstream tasks of 3D Visual Question Answering (3D-VQA) and 3D Situated Question Answering (3D-SQA). To this end, we develop novel multi-modal transformer-based architectures and we demonstrate how our pre-training method can benefit their performance. Quantitative and qualitative experimental results show that Multi-CLIP outperforms state-of-the-art works across the downstream tasks of 3D-VQA and 3D-SQA and leads to a well-structured 3D scene feature space.

Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes

TL;DR

The paper tackles how to transfer large-scale 2D vision-language knowledge to 3D scene reasoning. It introduces Multi-CLIP, a pre-training approach that aligns 3D scene representations with CLIP's text and multi-view image embeddings via a contrastive objective, aided by an auxiliary detection loss. Through pre-training a 3D scene encoder and transferring it to downstream 3D-VQA and 3D-SQA models, the method achieves state-of-the-art results on ScanQA and SQA3D and demonstrates a semantically structured 3D feature space. This work highlights the practical potential of cross-modal 2D-3D knowledge transfer for embodied 3D reasoning tasks.

Abstract

Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. However, it still remains understudied whether 2D distilled knowledge can provide useful representations for downstream 3D vision-language tasks such as 3D question answering. In this paper, we propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations. We leverage the representational power of the CLIP model by maximizing the agreement between the encoded 3D scene features and the corresponding 2D multi-view image and text embeddings in the CLIP space via a contrastive objective. To validate our approach, we consider the challenging downstream tasks of 3D Visual Question Answering (3D-VQA) and 3D Situated Question Answering (3D-SQA). To this end, we develop novel multi-modal transformer-based architectures and we demonstrate how our pre-training method can benefit their performance. Quantitative and qualitative experimental results show that Multi-CLIP outperforms state-of-the-art works across the downstream tasks of 3D-VQA and 3D-SQA and leads to a well-structured 3D scene feature space.
Paper Structure (12 sections, 3 equations, 4 figures, 4 tables)

This paper contains 12 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Pre-training Framework Overview
  5. Model Architecture for 3D-VQA
  6. Model Architecture for 3D-SQA
  7. Experiments
  8. Experimental setup
  9. Results
  10. Visualization
  11. Ablation study
  12. Conclusion

Figures (4)

  • Figure 1: Our pre-training method encourages the alignment of the 3D scene representation to the corresponding text and multi-view image embeddings in CLIP space via a contrastive loss.
  • Figure 2: The proposed model architectures for the two downstream tasks of 3D-VQA (left) and 3D-SQA (right).
  • Figure 3: Qualitative results on the ScanQA (left) and SQA3D dataset (right).
  • Figure 4: T-SNE visualizations of scene-level features in ScanNet. The 3D scene encoder weights learned during pre-training lead to a structured feature representation space.