Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

TL;DR

This work addresses the limited multi-view and 3D understanding in current LVLMs for autonomous driving by introducing NuInteract, a large-scale dataset with dense scene captions and 2D/3D annotations, and DriveMonkey, a plug-and-play framework that couples LVLMs with a spatial processing module via learnable queries. The spatial processor injects 3D priors from pre-trained detectors, enabling robust 3D visual grounding and diverse interactive driving tasks. Empirical results show DriveMonkey substantially outperforms generic LVLMs on NuInteract, especially in 3D VG, and benefits from dense caption pretraining. The approach offers a practical path to richer multimodal perception in autonomous driving with flexible integration of 3D detectors and LVLMs.

Abstract

The Large Visual-Language Models (LVLMs) have significantly advanced image understanding. Their comprehension and reasoning capabilities enable promising applications in autonomous driving scenarios. However, existing research typically focuses on front-view perspectives and partial objects within scenes, struggling to achieve comprehensive scene understanding. Meanwhile, existing LVLMs suffer from the lack of mapping relationship between 2D and 3D and insufficient integration of 3D object localization and instruction understanding. To tackle these limitations, we first introduce NuInteract, a large-scale dataset with over 1.5M multi-view image language pairs spanning dense scene captions and diverse interactive tasks. Furthermore, we propose DriveMonkey, a simple yet effective framework that seamlessly integrates LVLMs with a spatial processor using a series of learnable queries. The spatial processor, designed as a plug-and-play component, can be initialized with pre-trained 3D detectors to improve 3D perception. Our experiments show that DriveMonkey outperforms general LVLMs, especially achieving a 9.86% notable improvement on the 3D visual grounding task. The dataset and code will be released at https://github.com/zc-zhao/DriveMonkey.

Figures (6)

• Figure 1: (a) Comparison of existing datasets with NuInteract. While existing datasets use single-view images or focus on partial objects, NuInteract provides dense captions for the full scene and supports multi-object and 3D box information. (b) DriveMonkey framework utilizes multi-view images and user prompts to perform various interactive autonomous driving tasks.
• Figure 2: The pipeline of the NuInteract dataset annotation. VG denotes Visual Grounding. We collect various objects and their information from different experts, then filter them using Intersection over Union (IoU) and Image-Text Matching (ITM) criteria. The filtered objects and the corresponding information are then input into the Gemini team2023gemini to generate dense captions. We also use predefined templates combined with object information to create data for diverse interactive driving tasks.
• Figure 3: The statistical properties of the NuInteract dataset. VG refers to visual grounding. (a) The word frequency of dense captions. (b) The distribution of different interactive tasks. (c) The distribution of data volume under different camera views.
• Figure 4: The overall architecture of our DriveMonkey. The model first encodes input instruction prompts and multi-view images into embeddings. These embeddings and a group of learnable queries are fed into a Large Language Model (LLM). The output text tokens are used to generate associated language outputs. The spatial decoder receives the spatial feature from the spatial encoder, along with the learnable queries processed by the LLM, to detect the corresponding 3D object location.
• Figure 5: Comparison on all interactive tasks of using different data size for training. Avg. denotes the average of BLEU, ROUGE, and CIDEr. RD: Region Description; Pre: Prediction; VG: Visual Grounding.