Open 3D World in Autonomous Driving

TL;DR

Open3DWorld addresses open vocabulary perception in outdoor 3D autonomous driving by fusing LIDAR BEV features with textual inputs to locate objects described by text. It introduces a BEV-Region Text Fusion module and a CLIP-based text encoder integrated with an OpenSECOND BEV backbone, optimized with a contrastive and a localization loss. Evaluations on NuScenes-T and zero-shot Lyft Level 5 demonstrate the ability to adapt to unseen text without retraining while maintaining competitive 3D localization. The work advances multimodal perception for autonomous driving, enabling flexible, scalable recognition of novel categories in real-world urban environments.

Abstract

The capability for open vocabulary perception represents a significant advancement in autonomous driving systems, facilitating the comprehension and interpretation of a wide array of textual inputs in real-time. Despite extensive research in open vocabulary tasks within 2D computer vision, the application of such methodologies to 3D environments, particularly within large-scale outdoor contexts, remains relatively underdeveloped. This paper presents a novel approach that integrates 3D point cloud data, acquired from LIDAR sensors, with textual information. The primary focus is on the utilization of textual data to directly localize and identify objects within the autonomous driving context. We introduce an efficient framework for the fusion of bird's-eye view (BEV) region features with textual features, thereby enabling the system to seamlessly adapt to novel textual inputs and enhancing the robustness of open vocabulary detection tasks. The effectiveness of the proposed methodology is rigorously evaluated through extensive experimentation on the newly introduced NuScenes-T dataset, with additional validation of its zero-shot performance on the Lyft Level 5 dataset. This research makes a substantive contribution to the advancement of autonomous driving technologies by leveraging multimodal data to enhance open vocabulary perception in 3D environments, thereby pushing the boundaries of what is achievable in autonomous navigation and perception.

Figures (6)

• Figure 1: Unlike traditional closed-set 3D detection tasks, our Open3DWorld takes custom text as input and can locate and identify objects related to the text. Furthermore, new text inputs can be added seamlessly.
• Figure 2: The distribution of different texts in the NuScenes-T dataset, showing the number of objects described by each noun.
• Figure 3: Framework Overview: We start by separately extracting text features and bird's-eye view (BEV) features. Then, we use the BEV-Region Text Fusion module to align these features, connecting the BEV features with the text features. This alignment allows us to obtain unified BEV and text features. Finally, a contrastive head calculates the similarity between BEV region features and text features to identify the relevant positions in 3D space based on the text input. A localization head then refines the object's location and recognition details.
• Figure 4: We propose a progressive alignment module to fuse vanilla BEV features and vanilla text features. It takes the features of the two modalities, extracted separately by the backbone, as inputs and outputs Text-aware BEV feature and Updated text Feature, which provide a crucial basis for determining the direct connections between BEV grids and texts. Our fusion module fully leverages the attention mechanism's capability to integrate data from different modalities.
• Figure 5: NuScenes-T dataset’s results.The caption is the textual input. Based on the chosen textual data, Open3DWorld can output 3D boxes related to the text. Due to space limitations, additional visualization results are placed in the appendix. Note that our model takes only point cloud and text as input, with images used solely to display the detection results.