A Review of 3D Object Detection with Vision-Language Models

TL;DR

This paper surveys the field of 3D object detection through vision-language models (VLMs), comparing traditional geometry-based detectors with open-vocabulary, multimodal approaches. It synthesizes insights from over 100 papers, highlighting how VLMs enable semantic grounding, zero-shot reasoning, and instruction-driven perception in 3D space, while noting substantial trade-offs in computation, data requirements, and cross-modal alignment. The review details architectural foundations, pretraining/fine-tuning strategies, and visualization methods that ground language in 3D geometry, and it discusses current challenges such as limited 3D-language datasets, depth estimation in clutter, and real-time constraints. Collectively, the findings chart a path toward scalable, interpretable, and open-world 3D perception systems with broad implications for robotics, autonomous navigation, and human-robot interaction.

Abstract

This review provides a systematic analysis of comprehensive survey of 3D object detection with vision-language models(VLMs) , a rapidly advancing area at the intersection of 3D vision and multimodal AI. By examining over 100 research papers, we provide the first systematic analysis dedicated to 3D object detection with vision-language models. We begin by outlining the unique challenges of 3D object detection with vision-language models, emphasizing differences from 2D detection in spatial reasoning and data complexity. Traditional approaches using point clouds and voxel grids are compared to modern vision-language frameworks like CLIP and 3D LLMs, which enable open-vocabulary detection and zero-shot generalization. We review key architectures, pretraining strategies, and prompt engineering methods that align textual and 3D features for effective 3D object detection with vision-language models. Visualization examples and evaluation benchmarks are discussed to illustrate performance and behavior. Finally, we highlight current challenges, such as limited 3D-language datasets and computational demands, and propose future research directions to advance 3D object detection with vision-language models. >Object Detection, Vision-Language Models, Agents, VLMs, LLMs, AI

Figures (9)

• Figure 1: An illustration of VLM-based 3D detection on an apple, showing key advantages including semantic-awareness, zero-shot learning, multimodal fusion, human-aligned queries, and scalable data utilization.
• Figure 2: Temporal trend analysis highlighting the surge in global attention toward LLMs and VLMs. Following the public launch of ChatGPT on November 30, 2022, there has been a marked increase in interest and adoption of VLMs, particularly in domains such as 3D object detection, reflecting a shift toward multimodal and prompt-driven AI systems.
• Figure 3: Conceptual overview of the methodological and analytical framework used in this review. The mindmap illustrates the study's structure: starting with a comparison of 3D detection methods, followed by VLM-specific architectures, strengths and trade-offs, and ending with a discussion on challenges and future directions.
• Figure 4: Methodology diagram of this review paper illustrating the hybrid academic and AI-based search strategy, filtering process, and final paper selection, reducing 459 initial papers to 105.
• Figure 5: Illustration of a Vision-Language Model (VLM) performing multimodal reasoning—detecting, segmenting, and describing objects from images using textual prompts, demonstrating open-vocabulary 3D detection and contextual understanding.