Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu

TL;DR

3DMIT introduces an efficient alignment-free framework for 3D scene understanding by directly fusing 3D features with instruction prompts and fine-tuning via LoRA on representative LLM backbones. It builds a large 3D scene–language instruction dataset (75K pairs) derived from ScanNet using GPT-API, covering 3D VQA, captioning, grounding, and conversations, and includes a 3D grounding subset from ScanRefer. The method employs a 3D Perceiver that combines a frozen scene encoder with a 3D object encoder and an attributes encoder, then feeds the fused features into prompting-based fine-tuning, avoiding the traditional 3D–text alignment stage. Empirical results on 3D VQA and 3D grounding demonstrate competitive performance against alignment-based baselines while delivering improved efficiency; ablations reveal the benefits of multi-view prompts and the robustness across LLM types. The work advances practical, scalable 3D comprehension for LLMs and provides a public codebase for continued research and development.

Abstract

The remarkable potential of multi-modal large language models (MLLMs) in comprehending both vision and language information has been widely acknowledged. However, the scarcity of 3D scenes-language pairs in comparison to their 2D counterparts, coupled with the inadequacy of existing approaches in understanding of 3D scenes by LLMs, poses a significant challenge. In response, we collect and construct an extensive dataset comprising 75K instruction-response pairs tailored for 3D scenes. This dataset addresses tasks related to 3D VQA, 3D grounding, and 3D conversation. To further enhance the integration of 3D spatial information into LLMs, we introduce a novel and efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the alignment stage between 3D scenes and language and extends the instruction prompt with the 3D modality information including the entire scene and segmented objects. We evaluate the effectiveness of our method across diverse tasks in the 3D scene domain and find that our approach serves as a strategic means to enrich LLMs' comprehension of the 3D world. Our code is available at https://github.com/staymylove/3DMIT.

3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

TL;DR

3DMIT introduces an efficient alignment-free framework for 3D scene understanding by directly fusing 3D features with instruction prompts and fine-tuning via LoRA on representative LLM backbones. It builds a large 3D scene–language instruction dataset (75K pairs) derived from ScanNet using GPT-API, covering 3D VQA, captioning, grounding, and conversations, and includes a 3D grounding subset from ScanRefer. The method employs a 3D Perceiver that combines a frozen scene encoder with a 3D object encoder and an attributes encoder, then feeds the fused features into prompting-based fine-tuning, avoiding the traditional 3D–text alignment stage. Empirical results on 3D VQA and 3D grounding demonstrate competitive performance against alignment-based baselines while delivering improved efficiency; ablations reveal the benefits of multi-view prompts and the robustness across LLM types. The work advances practical, scalable 3D comprehension for LLMs and provides a public codebase for continued research and development.

Abstract

The remarkable potential of multi-modal large language models (MLLMs) in comprehending both vision and language information has been widely acknowledged. However, the scarcity of 3D scenes-language pairs in comparison to their 2D counterparts, coupled with the inadequacy of existing approaches in understanding of 3D scenes by LLMs, poses a significant challenge. In response, we collect and construct an extensive dataset comprising 75K instruction-response pairs tailored for 3D scenes. This dataset addresses tasks related to 3D VQA, 3D grounding, and 3D conversation. To further enhance the integration of 3D spatial information into LLMs, we introduce a novel and efficient prompt tuning paradigm, 3DMIT. This paradigm eliminates the alignment stage between 3D scenes and language and extends the instruction prompt with the 3D modality information including the entire scene and segmented objects. We evaluate the effectiveness of our method across diverse tasks in the 3D scene domain and find that our approach serves as a strategic means to enrich LLMs' comprehension of the 3D world. Our code is available at https://github.com/staymylove/3DMIT.
Paper Structure (30 sections, 5 figures, 6 tables)

This paper contains 30 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. 3D Scene Understanding
  4. 3D Multi-modal Large Language Models
  5. Dataset
  6. 3D VQA, Captioning and Conversations
  7. 3D Grounding
  8. Method
  9. Overview
  10. 3D Perceiver
  11. 3D Scene Encoder
  12. 3D Object Encoder
  13. Instruction Tuning
  14. Prompt
  15. Pre-trained LLM and Fine-tune
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Overview Architecture of our method. The 3D input covers 3D scene and segmented objects. Our 3D perceiver contains frozen 3D scene encoder and 3D object encoder, trainable attributes encoder and projectors. We construct 3D vision-language prompt with system message, 3D features and task instructions. Leveraging the 3D prompt, we can fine-tune LLM efficiently by Lora to solve 3D tasks.
  • Figure 2: The case study of 3D conversation and 3D caption task.
  • Figure 3: The case study of 3D conversation and 3D VQA task.
  • Figure 4: The 3D vision-language prompt for VQA with system message, 3D features and task instructions.
  • Figure 5: The 3D vision-language prompt for Visual grounding with system message, 3D features and task instructions.