Talk to Parallel LiDARs: A Human-LiDAR Interaction Method Based on 3D Visual Grounding

TL;DR

This work extends autonomous driving perception by introducing 3D visual grounding into parallel LiDARs, enabling human–LiDAR interaction. It introduces Talk2LiDAR, a large-scale outdoor benchmark built on nuScenes, and BEVGrounding, a one-stage fusion model that tightly aligns text, images, and LiDAR point clouds in BEV space. Through experiments on Talk2Car-3D and Talk2LiDAR, BEVGrounding achieves strong gains over two-stage baselines, with ablations highlighting the value of CLIP-based language encoding and the proposed fusion modules. The results underscore the potential for cognitive, interactive LiDAR systems and pave the way for integrating multimodal foundation models in realistic driving scenarios.

Abstract

LiDAR sensors play a crucial role in various applications, especially in autonomous driving. Current research primarily focuses on optimizing perceptual models with point cloud data as input, while the exploration of deeper cognitive intelligence remains relatively limited. To address this challenge, parallel LiDARs have emerged as a novel theoretical framework for the next-generation intelligent LiDAR systems, which tightly integrate physical, digital, and social systems. To endow LiDAR systems with cognitive capabilities, we introduce the 3D visual grounding task into parallel LiDARs and present a novel human-computer interaction paradigm for LiDAR systems. We propose Talk2LiDAR, a large-scale benchmark dataset tailored for 3D visual grounding in autonomous driving. Additionally, we present a two-stage baseline approach and an efficient one-stage method named BEVGrounding, which significantly improves grounding accuracy by fusing coarse-grained sentence and fine-grained word embeddings with visual features. Our experiments on Talk2Car-3D and Talk2LiDAR datasets demonstrate the superior performance of BEVGrounding, laying a foundation for further research in this domain.

Figures (7)

• Figure 1: Introduction for the visual grounding task in autonomous driving. 2D visual grounding utilizes an image and language prompt as input (Fig.1a), while 3D visual grounding utilizes multi-view images, point clouds, and prompts as input (Fig.1b).
• Figure 2: The architecture of the two-stage baseline method.
• Figure 3: The network architecture of our proposed one-stage BEVGrounding method.
• Figure 4: Visualization results of the two-stage baseline and one-stage BEVGrounding method. Red, blue, and green boxes denote the ground truth, predicted boxes by the baseline, and predicted boxes by BEVGrounding, respectively.
• Figure 5: Word cloud of language prompts from the Talk2LiDAR dataset.