Sce2DriveX: A Generalized MLLM Framework for Scene-to-Drive Learning

TL;DR

The paper tackles the challenge of generalizing end-to-end autonomous driving across diverse scenes while aligning model reasoning with human driving. It introduces Sce2DriveX, a generalized multimodal large language model framework that performs multimodal joint learning from local scene videos and global BEV maps, guided by a chain-of-thought style reasoning process. It contributes a first comprehensive VQA driving instruction dataset with hierarchical scene understanding and interpretable end-to-end driving, along with a task-oriented three-stage training pipeline. Experimental results on nuScenes and CARLA Bench2Drive show state-of-the-art performance across scene understanding, meta-action reasoning, behavior interpretation, motion planning, and control signal generation, with robust cross-scene generalization and improved interpretability.

Abstract

End-to-end autonomous driving, which directly maps raw sensor inputs to low-level vehicle controls, is an important part of Embodied AI. Despite successes in applying Multimodal Large Language Models (MLLMs) for high-level traffic scene semantic understanding, it remains challenging to effectively translate these conceptual semantics understandings into low-level motion control commands and achieve generalization and consensus in cross-scene driving. We introduce Sce2DriveX, a human-like driving chain-of-thought (CoT) reasoning MLLM framework. Sce2DriveX utilizes multimodal joint learning from local scene videos and global BEV maps to deeply understand long-range spatiotemporal relationships and road topology, enhancing its comprehensive perception and reasoning capabilities in 3D dynamic/static scenes and achieving driving generalization across scenes. Building on this, it reconstructs the implicit cognitive chain inherent in human driving, covering scene understanding, meta-action reasoning, behavior interpretation analysis, motion planning and control, thereby further bridging the gap between autonomous driving and human thought processes. To elevate model performance, we have developed the first extensive Visual Question Answering (VQA) driving instruction dataset tailored for 3D spatial understanding and long-axis task reasoning. Extensive experiments demonstrate that Sce2DriveX achieves state-of-the-art performance from scene understanding to end-to-end driving, as well as robust generalization on the CARLA Bench2Drive benchmark.

Figures (5)

• Figure 1: Overall pipeline. Our goal is to achieve generalization and consensus in cross-scene driving, including three novel contributions: 1) Sce2DriveX framework; 2) comprehensive VQA driving instruction dataset; and 3) three-stage training pipeline.
• Figure 2: Model architecture. Sce2DriveX uses modal encoders to emergently align the visual representations of multi-view scene videos and BEV map images into a unified visual feature space, which are then mapped to the LLM backbone through a shared projection.
• Figure 3: Dataset construction. Comprehensive VQA driving instruction dataset consists of two subsets: 1) Hierarchical Scene Understanding Dataset; 2) Interpretable End-to-End Driving Dataset.
• Figure 4: Visualization results of generalization testing (corner cases from the driving simulation dataset Bench2Drive).
• Figure 5: Qualitative demonstration of Sce2DriveX.