3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding

TL;DR

This paper tackles the challenge of 3D scene understanding with multimodal large language models by introducing 3UR-LLM, an end-to-end framework that ingests 3D point clouds, compresses 3D features into a compact set of tokens, and fuses them with text through a 3D query fusion mechanism. It also presents 3DS-160K, a high-quality 3D-text dataset (160K samples) generated via open-source 2D MLLMs and LLMs to enable pre-training across 3D tasks such as dense captioning and visual question answering. Empirical results show that 3UR-LLM outperforms prior 3D LLMs on ScanQA and SQA3D benchmarks, with notable efficiency gains (e.g., substantial reductions in training time) and robust ablation evidence for the importance of the 3D compressor and 3D query fusion components. The work provides a scalable, end-to-end pipeline for 3D multimodal understanding and demonstrates the value of a dedicated 3D-text dataset to advance embodied AI in real-world environments.

Abstract

Multi-modal Large Language Models (MLLMs) exhibit impressive capabilities in 2D tasks, yet encounter challenges in discerning the spatial positions, interrelations, and causal logic in scenes when transitioning from 2D to 3D representations. We find that the limitations mainly lie in: i) the high annotation cost restricting the scale-up of volumes of 3D scene data, and ii) the lack of a straightforward and effective way to perceive 3D information which results in prolonged training durations and complicates the streamlined framework. To this end, we develop pipeline based on open-source 2D MLLMs and LLMs to generate high-quality 3D-text pairs and construct 3DS-160K , to enhance the pre-training process. Leveraging this high-quality pre-training data, we introduce the 3UR-LLM model, an end-to-end 3D MLLM designed for precise interpretation of 3D scenes, showcasing exceptional capability in navigating the complexities of the physical world. 3UR-LLM directly receives 3D point cloud as input and project 3D features fused with text instructions into a manageable set of tokens. Considering the computation burden derived from these hybrid tokens, we design a 3D compressor module to cohesively compress the 3D spatial cues and textual narrative. 3UR-LLM achieves promising performance with respect to the previous SOTAs, for instance, 3UR-LLM exceeds its counterparts by 7.1\% CIDEr on ScanQA, while utilizing fewer training resources. The code and model weights for 3UR-LLM and the 3DS-160K benchmark are available at 3UR-LLM.

Figures (9)

• Figure 1: Demonstration of 3UR-LLM, a multi-modal large language model tailored for comprehending 3D scenes, which enables flexible and precise responses to given instructions, showing robust reasoning capabilities in real-world environments.
• Figure 2: (a) Task distribution across different categories. (b) Token frequency distribution in answers and (c) questions for the 3DS-160K dataset. The x-axis represents the number of tokens, and the y-axis denotes their frequency. Tokens are generated using the T5 tokenizer raffel2020exploring.
• Figure 3: The overview of the proposed data generation pipeline for 3DS-160K. The conceptual workflow mainly consists of three steps: 1) multi-view captions generation. 2) question-answer pairs and scene captions generation. 3) sample pairs refinement.
• Figure 4: The pipeline of our 3UR-LLM framework. (a) Overview of the pipeline. The 3D compressor consolidates scene features and textual tokens into unified queries, which are then fed into a large language model to generate textual responses. (b) Details of the 3D compressor which condensed 3D features and textual tokens into compact queries.
• Figure 5: Word clouds depicting the distribution of words in answers (a) and questions (b) within our 3DS-160K dataset.