MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

TL;DR

MiniGPT-3D tackles the high cost of aligning 3D point clouds with LLMs by leveraging 2D vision-language priors from 2D-LLMs. It introduces a two-step projection (3D to 2D to 1D) architecture, a four-stage cascaded training strategy, and a lightweight Mixture of Query Experts to adaptively fuse multi-perspective 3D features into the 2D-LLM semantic space, all while applying parameter-efficient fine-tuning. With only 47.8M trainable parameters and 26.8 hours of training on a single RTX 3090, MiniGPT-3D achieves state-of-the-art results in generative 3D object classification and captioning, outperforming larger baselines by substantial margins. The approach democratizes 3D-language understanding by drastically reducing computational requirements, offering practical utility for robotics, surveying, and autonomous systems, and sets a foundation for further 2D-driven 3D-LLM development.

Abstract

Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. Code and weights are available at https://github.com/TangYuan96/MiniGPT-3D.

Figures (8)

• Figure 1: Existing methods and ours to align 3D with LLMs.
• Figure 2: Training framework and strategy. Our MiniGPT-3D utilizes a four-stage training strategy. (a) We solely train the point cloud projection layer (MLP). (b) We train the modality projector while fine-tuning the point cloud projection layer, Q-Former, and LLM backbone. (c) We further enhance the modules trained in the second stage by leveraging a more challenging task. (d) Finally, we only train the mixture of query experts, while freezing the remaining modules.
• Figure 3: The framework of the mixture of query experts. First, a point cloud is encoded to features $X$ and $Y$. Feature $X$ is then passed through to the expert router, assigning softmax-based weights to experts. The top $g$ experts are selected based on these weights. These experts, together with $Y$, are then fed into the Q-Former, and their outputs are weighted to produce the final point queries $\overline{Q}$.
• Figure 4: 3D recognition and captioning results (#1) of our MiniGPT-3D. Given a 3D point cloud and a corresponding question, MiniGPT-3D generates a text response. The correct and fine-grained answers are highlight in green.
• Figure 5: 3D recognition and captioning results (#2) of our MiniGPT-3D. Given a 3D point cloud and a corresponding question, MiniGPT-3D generates a text response. The correct and fine-grained answers are highlight in green.