MoniRefer: A Real-world Large-scale Multi-modal Dataset based on Roadside Infrastructure for 3D Visual Grounding

TL;DR

This work introduces 3D visual grounding for outdoor monitoring by leveraging roadside infrastructure sensors. It presents MoniRefer, the first large-scale real-world multi-modal dataset for roadside-level 3D grounding, and Moni3DVG, an end-to-end framework that fuses LiDAR geometry, image appearance, and language to localize referred objects in 3D space. The dataset comprises 136,018 objects and 411,128 natural language descriptions across diverse intersections, with rigorous annotation and privacy safeguards. Empirical results show state-of-the-art performance over baselines and strong ablations demonstrate the value of multi-modal fusion, indicating significant potential for intelligent infrastructure and smart city applications.

Abstract

3D visual grounding aims to localize the object in 3D point cloud scenes that semantically corresponds to given natural language sentences. It is very critical for roadside infrastructure system to interpret natural languages and localize relevant target objects in complex traffic environments. However, most existing datasets and approaches for 3D visual grounding focus on the indoor and outdoor driving scenes, outdoor monitoring scenarios remain unexplored due to scarcity of paired point cloud-text data captured by roadside infrastructure sensors. In this paper, we introduce a novel task of 3D Visual Grounding for Outdoor Monitoring Scenarios, which enables infrastructure-level understanding of traffic scenes beyond the ego-vehicle perspective. To support this task, we construct MoniRefer, the first real-world large-scale multi-modal dataset for roadside-level 3D visual grounding. The dataset consists of about 136,018 objects with 411,128 natural language expressions collected from multiple complex traffic intersections in the real-world environments. To ensure the quality and accuracy of the dataset, we manually verified all linguistic descriptions and 3D labels for objects. Additionally, we also propose a new end-to-end method, named Moni3DVG, which utilizes the rich appearance information provided by images and geometry and optical information from point cloud for multi-modal feature learning and 3D object localization. Extensive experiments and ablation studies on the proposed benchmarks demonstrate the superiority and effectiveness of our method. Our dataset and code will be released.

Figures (8)

• Figure 1: Introduction to 3D Visual Grounding for Outdoor Monitoring Scenarios. Moni3DVG integrates multi-modal visual data collected from roadside infrastructure sensors with natural language descriptions to localize the referred object in highly dynamic and complex traffic environments. For clearer visualization, the localization results are also presented in the corresponding images.
• Figure 2: Our data collection pipeline. For instance bounding boxes, we employ a pretrained object detector to generate preliminary annotations of objects, followed by manual inspection and refinement. For natural language expressions, annotators are required to provide diverse attributes for each object while labeling the 3D bounding boxes. Then, we obtain semantically rich and diverse textual descriptions by combining manual annotation with LLM-assisted generation.
• Figure 3: Sensors Setup and Annotation Area. (a) showcases the sensor layout of multi-source data acquisition platform. (b) demonstrates the effective annotation range composed of four sensors.
• Figure 4: (a)-(d) present word clouds of terms corresponding to unique words across the dataset, color, coarse-grained object categories, and fine-grained object categories in the MoniRefer dataset, respectively. The fonts size reflects the occurrence frequency of each term in the descriptions.
• Figure 5: The overview of Moni3DVG framework. We formulate the 3DVG task as identifying the candidate point that is closest to the target object center. Specifically, the visual and linguistic encoders extract candidate point representations and textual features from the point clouds and natural language description, respectively. These features are projected into a latent space, where effective cross-modal interaction and fusion are performed to yield enriched multi-modal representations. Finally, the spatial localization module leverages the fused multi-modal features to identify the candidate point nearest to the referred object center and output the 3D bounding box that best matches the referring expression.