Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy

TL;DR

This work surveys the convergence of Large Language Models with 3D vision to enable intelligent robotic perception and autonomy. It analyzes foundational LLM and 3D-vision principles, surveys sensing modalities, and details state-of-the-art methods across grounding, dynamic scene understanding, open-vocabulary learning, text-to-3D generation, multimodal fusion, and embodied agents. The review also catalogs robotics-focused datasets and evaluation metrics, discusses practical constraints such as computational demands and data scarcity, and highlights challenges in safety and real-time deployment. By outlining architectural paradigms, multimodal fusion strategies and open-vocabulary approaches, the paper articulates a roadmap for robust, context-aware, autonomous robotic systems that can reason over 3D environments using language.

Abstract

With the rapid advancement of artificial intelligence and robotics, the integration of Large Language Models (LLMs) with 3D vision is emerging as a transformative approach to enhancing robotic sensing technologies. This convergence enables machines to perceive, reason and interact with complex environments through natural language and spatial understanding, bridging the gap between linguistic intelligence and spatial perception. This review provides a comprehensive analysis of state-of-the-art methodologies, applications and challenges at the intersection of LLMs and 3D vision, with a focus on next-generation robotic sensing technologies. We first introduce the foundational principles of LLMs and 3D data representations, followed by an in-depth examination of 3D sensing technologies critical for robotics. The review then explores key advancements in scene understanding, text-to-3D generation, object grounding and embodied agents, highlighting cutting-edge techniques such as zero-shot 3D segmentation, dynamic scene synthesis and language-guided manipulation. Furthermore, we discuss multimodal LLMs that integrate 3D data with touch, auditory and thermal inputs, enhancing environmental comprehension and robotic decision-making. To support future research, we catalog benchmark datasets and evaluation metrics tailored for 3D-language and vision tasks. Finally, we identify key challenges and future research directions, including adaptive model architectures, enhanced cross-modal alignment and real-time processing capabilities, which pave the way for more intelligent, context-aware and autonomous robotic sensing systems.

Figures (15)

• Figure 1: Major LLM Development: Various different LLM models (BERT devlin2018bert, ALBERT lan2019albert, DistilBERT sanh2019distilbert, RoBERTa liu2019roberta, SpanBERT joshi2020spanbert, GPT Radford2018ImprovingLU, GPT-2 Radford2019LanguageMA, GPT-3 brown2020language, GPT-4 achiam2023gpt, Llama 3.1 Llama3.1, InstructGPT InstructGPT, BART BART, DeepSeek R1 DeepSeekR1), along with their parameter count and key feature of improvement. 1 GPT-4 parameters count source: GPT4Wikipedia.
• Figure 2: Transformer Architecture: Constructed on an encoder-decoder framework, the architecture employs multiple linear, normalization, embedding, feed-forward, self-attention and cross-attention blocks to generate outputs efficiently AttentionIsAllYouNeed.
• Figure 3: Categories of 3D Data Representations: 3D data can be classified into two primary categories: Euclidean (including Descriptors, Projections, RGB-D, Volumetric and Multi-view) and Non-Euclidean (comprising Point Clouds, Graphs and Meshes) 3dDataRepresentationsSurvey.
• Figure 4: Research areas at the intersection of 3D and LLMs: Advancing capabilities in 3D scene understanding, generation and modification; object grounding and referencing; developing embodied agents; and integrating 3D data with multimodal language models.
• Figure 5: Overview of different 3D vision sensing techniques work: stereo vision (a),structured light (b) and time of flight (c), illustrating their working principles and components for capturing spatial information.