RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving

TL;DR

RoboDriveBench provides a robustness benchmark for VLM-driven autonomous driving, highlighting vulnerabilities to prompt corruption and the need for multimodal fusion. The authors propose RoboDriveVLM, a VLM-based end-to-end framework that fuses LiDAR, radar, and camera data, plus a test-time adaptation (TTA) via cross-modal knowledge distillation to improve resilience. Across sensor and prompt corruptions, RoboDriveVLM (especially with TTA) achieves superior trajectory accuracy and safety metrics, demonstrating the importance of multimodal fusion and online adaptation. The work delivers practical contributions including a new benchmark, a robust model, and a reproducible evaluation methodology with detailed simulation and reproduction details.

Abstract

Current Vision-Language Model (VLM)-based end-to-end autonomous driving systems often leverage large language models to generate driving decisions directly based on their understanding of the current scene. However, such systems introduce multiple risks in real-world driving scenarios. To evaluate whether VLMs are truly viable for autonomous driving, we introduce RoboDriveBench, the first robustness benchmark focused on end-to-end trajectory prediction tasks. This benchmark systematically evaluates two critical categories of real-world challenges for VLM-based end-to-end autonomous driving systems through 11 simulated scenarios encompassing various corruption types, including 6 scenarios of sensor corruption caused by environmental variations, along with 5 cases of prompt corruption resulting from human intervention and data transmission failures. Each corruption type includes 250 unique driving scenarios and 5,689 frames, resulting in 64,559 total trajectory prediction cases per evaluation. To overcome these real-world challenges, we propose a novel VLM-based autonomous driving framework called RoboDriveVLM, which enhances robustness by mapping more multimodal data-e.g., lidar and radar-into a unified latent space. Furthermore, we introduce a new Test-Time Adaptation (TTA) method based on cross-modal knowledge distillation to improve the robustness of VLM-based autonomous driving systems. Through extensive experiments, our work highlights the limitations of current VLM-based end-to-end autonomous driving systems and provides a more reliable solution for real-world deployment. Source code and datasets will be released.

Figures (6)

• Figure 1: RoboDriveBench targets trajectory prediction tasks in real-world driving scenarios and provides a systematic evaluation of VLM-based end-to-end autonomous driving models under two major categories of typical corruptions: sensor corruption and prompt corruption. Moreover, considering the tendency of VLMs to produce invalid outputs during trajectory point generation, the benchmark introduces a novel evaluation metric designed to effectively quantify such invalid predictions.A single test comprises 64,559 individual trajectory prediction tasks, covering 11 specific types of corruption, each with 250 scenarios and 5,689 frames of data.
• Figure 2: Method Overview. Module (a) present our VLM-based end-to-end autonomous driving system. We map the point cloud data from LiDAR and radar, along with six-view camera images, into a unified image coordinate system. The framework primarily relies on LiDAR for spatial structure, with cameras complementing semantic information and radar providing velocity information of surrounding objects. Module (b) present our test-time adaptation (TTA) based cross-modal knowledge distillation approach. We perform trajectory distillation across different modalities to achieve modality decoupling during inference, thereby enhancing the model's robustness in complex autonomous driving scenarios.
• Figure 3: Radar chart comparison of model performance across four evaluation metrics: L2, Collision (Col), MCL2, and MCC, each under both sensor-based and prompt-based input settings.
• Figure 4: Examples of six sensor corruption methods at severity levels 1, 3, and 5.
• Figure 5: Examples of five prompt corruption methods