WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language

TL;DR

This work addresses $3D$ Visual Grounding in the Wild ($3DVGW$), where targets must be localized in 3D space from online multi-modal sensor data and natural language. It introduces WildRefer, a one-stage framework that combines a Dynamic Visual Encoder (DVE) with Triple-modal Feature Interaction (TFI) and a DETR-like decoder to fuse LiDAR geometry, camera appearance, and language cues for accurate 3D grounding. The authors also present two large-scale, human-centric datasets, STRefer and LifeRefer, captured with synchronized LiDAR and cameras to support dynamic scene grounding with free-form descriptions. Empirical results show state-of-the-art performance on both datasets, validating the importance of temporal dynamics and cross-modal fusion for robust grounding in wild, large-scale environments, with clear implications for autonomous driving and service robotics.

Abstract

We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, dubbed WildRefer, for this task by fully utilizing the rich appearance information in images, the position and geometric clues in point cloud as well as the semantic knowledge of language descriptions. Besides, we propose two novel datasets, i.e., STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios accompanied with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive experiments and ablation studies demonstrate that our method achieves state-of-the-art performance on the proposed benchmarks. The code is provided in https://github.com/4DVLab/WildRefer.

Figures (5)

• Figure 2: (a), (b), (c) and (d), (e), (f) are word clouds for action words, non-human object words, and spatial relation words on STRefer and LifeRefer, respectively.
• Figure 3: Pipeline of WildRefer. The inputs are multi-frame synchronized points clouds and images as well as a natural language description. After feature extraction, we obtain two types of visual feature and a text feature. Through two dynamic visual encoders, we extract dynamic-enhanced image and point features. Then, a triple-modal feature interaction module is followed to fuse valuable information from different modalities. Finally, through a DETR-like decoder, we decode the location and size of the target object. SA and CA denote self-attention and cross-attention, respectively.
• Figure 4: Structure of dynamic visual encoder. $F$ indicates the feature extracted from either point clouds or images, $t$ is the time, and PosEmb means position embedding.
• Figure 5: Visualization of comparison results on LifeRefer. Blue, red, and green boxes denote the ground truth, wrong prediction, and right prediction, respectively.
• Figure 6: Visualization results of ablation results on STRefer.