LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

TL;DR

LL3DA introduces a large language model–driven agent that directly consumes 3D point clouds and optional visual prompts to perform understanding, reasoning, and planning in diverse 3D environments. It fuses textual instructions, visual prompts, and 3D geometry via a frozen 3D encoder, a visual prompt module, and a Q-Former to produce a fixed-length token prefix that conditions a frozen LLM for autoregressive generation. The approach achieves state-of-the-art results on 3D Dense Captioning and 3D Question Answering on ScanRefer, Nr3D, and ScanQA, and benefits from visual prompts to disambiguate cluttered scenes. Extensive ablations and qualitative analyses support the design choices (early fusion of prompts, Q-Former interactions) and demonstrate LL3DA’s potential as a generalist agent for scene description, QA, and embodied planning in 3D. This work advances practical 3D vision–language systems by enabling direct 3D input processing with LLM-based instruction following and planning capabilities.

Abstract

Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.

Figures (9)

• Figure 1: Overview of the Proposed Approach. (a) The overall pipeline of our proposed LL3DA first extracts interaction-aware 3D scene embeddings, which are later projected to the prefix of textual instructions as the input of a frozen LLM. (b) The detailed design of the Interactor3D, which aggregates visual prompts, textual instructions, and 3D scene embeddings into a fixed length querying tokens. (c) The prompt encoder encodes the user clicks and box coordinates with the positional embeddings and ROI features, respectively.
• Figure 2: Different Ways of Encoding Visual Prompts. We listed two ways of encoding visual prompts, (a) adopting a unified transformer to aggregate features from all kinds of interactions, and (b) directly concatenate the visual prompts to the scene embeddings. Experiments (\ref{['tab:ablation-qformer-design']}) show that early fusion(a) leads to a better performance.
• Figure 3: Qualitative Results. We provide several visualization results on various 3D vision and language tasks in diverse 3D environments (living room, classroom, kitchen, and bedroom). RGB]255, 124, 128Red highlights the wrong answer.
• Figure 4: Qualitative Results on Scene Descriptions. We highlight some of the phrases in the generated scene descriptions mentioning the instances in the 3D environment.
• Figure 5: More Qualitative Results on 3D Dense Captioning (upper), 3D Question Answering (middle), and Embodied Dialogue (lower). RGB]255, 124, 128Red highlights the wrong answer.