Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models

TL;DR

This work addresses the gap in holistic language-based autonomous driving by introducing NuInstruct, a $91{,}355$-pair multi-view video QA dataset spanning $17$ subtasks that require temporal, multi-view, and spatial reasoning. It proposes a SQL-based data-generation pipeline to automate instruction creation aligned with human driving progression and develops BEV-InMLLM, an end-to-end framework that injects BEV features into multimodal LLMs via a plug-and-play BEV-Injection module. The approach combines a Multi-View LLM (MV-MLLM) with a BEV-aware fusion strategy, achieving notable improvements over strong baselines (approximately $9\%$ overall) and demonstrating the importance of temporal, multi-view, and BEV information for robust driving understanding. By enabling instruction-aware BEV fusion and scalable data generation, this work offers a practical path toward more capable language-guided autonomous driving systems and sets the stage for future enhancements involving traffic signals and 3D object detection.

Abstract

The rise of multimodal large language models (MLLMs) has spurred interest in language-based driving tasks. However, existing research typically focuses on limited tasks and often omits key multi-view and temporal information which is crucial for robust autonomous driving. To bridge these gaps, we introduce NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17 subtasks, where each task demands holistic information (e.g., temporal, multi-view, and spatial), significantly elevating the challenge level. To obtain NuInstruct, we propose a novel SQL-based method to generate instruction-response pairs automatically, which is inspired by the driving logical progression of humans. We further present BEV-InMLLM, an end-to-end method for efficiently deriving instruction-aware Bird's-Eye-View (BEV) features, language-aligned for large language models. BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g. around 9% improvement on various tasks. We plan to release our NuInstruct for future research development.

Figures (12)

• Figure 1: Example of our proposed NuInstruct dataset for holistic language-based autonomous driving.(a) The input are multi-view videos. (b) Various tasks are presented in instruction-response format. There are a total of four tasks, covering $17$ subtasks (see in Fig.\ref{['fig:statistics']} (a)).
• Figure 2: Procedure of SQL-based data generation. We formulate the data generation into an SQL-based process, using different task SQLs to retrieve the response from the scene information database. The design of SQLs follows the logical flow of autonomous driving tasks hu2023planning, which is represented in blue dashed arrows. 'Planning w/ R' indicates the planning with reasoning.
• Figure 3: The illustration of an example for Step 3 retrieval in the data generation process.(a) Sampled keyframes with annotations. Three keyframes with annotations are randomly sampled, and we only select one instance, i.e., the pedestrian (box), in this example for clarity. (b) Sampled subtask SQLs. Each subtask SQL consists of two parts, i.e., the subtask function and the instruction prompt. (c) Retrieved Responses. The subtask function receives the specific input and retrieves the responses from the scene information database.
• Figure 4: Statistics of NuInstruct. (a) Proportions of different tasks. The size of the arc represents the proportions of each task, while the same color indicates tasks of the same category. Our task encompasses a diverse range of tasks including perception, prediction, risk, and planning. (b) Response numbers under different views. The horizontal axis represents different views, and the vertical axis indicates the number of responses requiring information from the corresponding view. (c) View percentages within different tasks. The horizontal and vertical axes represent the proportion of different views and task classes, respectively.
• Figure 5: The overall pipeline of our proposed BEV-InMLLM.(a) The base multimodal large language model (MLLM) tailored for processing the multi-view videos. (b) The bird's-eye-view injection module (BEV-In) to inject BEV representations into base MLLM to boost autonomous driving understanding.