DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning

TL;DR

DSDrive presents a lightweight end-to-end autonomous driving framework that distills high-level reasoning from a large Vision-Language Model into a compact LLM and unifies reasoning with planning through a waypoint-driven dual-head coordination. By externalizing reasoning via a think-and-answer dataset and tying it to waypoint planning, DSDrive achieves competitive closed-loop performance in CARLA while reducing inference time and memory usage. The approach combines a VLM-based reasoning model with a compact driving model, enabling interpretable decisions and efficient deployment on resource-constrained platforms. Extensive experiments and qualitative analyses show that the dual-head coordination effectively aligns reasoning and planning, offering practical benefits for robust, explainable autonomous driving in real-world scenarios.

Abstract

We present DSDrive, a streamlined end-to-end paradigm tailored for integrating the reasoning and planning of autonomous vehicles into a unified framework. DSDrive leverages a compact LLM that employs a distillation method to preserve the enhanced reasoning capabilities of a larger-sized vision language model (VLM). To effectively align the reasoning and planning tasks, a waypoint-driven dual-head coordination module is further developed, which synchronizes dataset structures, optimization objectives, and the learning process. By integrating these tasks into a unified framework, DSDrive anchors on the planning results while incorporating detailed reasoning insights, thereby enhancing the interpretability and reliability of the end-to-end pipeline. DSDrive has been thoroughly tested in closed-loop simulations, where it performs on par with benchmark models and even outperforms in many key metrics, all while being more compact in size. Additionally, the computational efficiency of DSDrive (as reflected in its time and memory requirements during inference) has been significantly enhanced. Evidently thus, this work brings promising aspects and underscores the potential of lightweight systems in delivering interpretable and efficient solutions for AD.

Figures (8)

• Figure 1: The dataset for distillation is prepared with an explicit think-and-answer process. This design enables the model to learn complex reasoning steps before generating a final answer, enhancing its learning of nuanced driving tasks. The dataset is diverse and realistic, covering various driving scenarios to improve model robustness and generalization.
• Figure 2: The proposed E2E driving model boasts advanced reasoning and planning capabilities. Its student model ensures inference efficiency and is enhanced via distillation from a teacher model, which substantially improves reasoning skills. By integrating reasoning and planning, it achieves robust performance and interpretability, making it highly reliable and accurate for AD tasks.
• Figure 3: The explicit think-and-answer reasoning process for representative driving scenarios from the CARLA simulator. This includes urban road and highway settings, as well as daytime and nighttime conditions.
• Figure 4: Performance at a typical urban intersection in response to traffic light signals. The AV first stops at a red light and subsequently starts driving when the light turns green (left). The visualizations of the throttle and brake control inputs of the AV (right) demonstrate its prompt deceleration and acceleration in accordance with the traffic signal phase.
• Figure 5: Performance when the AV approaches an intersection and makes a left turn. The image (top left) shows the predicted waypoints from the prediction heads in planning (red dots) and reasoning (cyan dots). The visualizations of the steering (bottom left) and BEV trajectory (right) are included to validate the efficacy of closed-loop planning and control.

Theorems & Definitions (1)

• Remark 1