LMDrive: Closed-Loop End-to-End Driving with Large Language Models

TL;DR

This work tackles the challenge of enabling language-guided, closed-loop end-to-end autonomous driving by integrating a frozen large language model with a multi-view vision encoder to interpret natural language instructions alongside sensor data. It introduces a CARLA-based dataset of ~64K instruction-following clips and the LangAuto benchmark to evaluate language-conditioned driving in diverse, realistic scenarios. The LMDrive architecture couples a BEV-based vision encoder with a Q-Former–assisted LLM backbone (LLaMA) using adapters, trained in a two-stage process to predict control signals and instruction completion, augmented by PID control for low-level actuation. Empirical results and ablations demonstrate the value of perception pre-training, BEV token integration, and effective token fusion for language-guided driving, and the LangAuto benchmark reveals practical benefits of including narrative instructions and safety notices for robust interaction with humans and navigation software.

Abstract

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

Figures (8)

• Figure 1: We present LMDrive, the first language-guided closed-loop end-to-end driving framework. LMDrive takes as input the language instruction and multi-modal multi-view sensor data, and outputs control signals in real-time to drive in complex scenarios.
• Figure 2: Two examples of the collected data with corresponding labeled navigation instructions and optional notice instructions.
• Figure 3: Distribution of parsed clips in terms of clip length and the corresponding navigation instruction type.
• Figure 4: The structure of the proposed LMDrive model, which consists of two major components: 1) a vision encoder that processes multi-view multi-modal sensor data (camera and LiDAR) for scene understanding and generating visual tokens; 2) a large language model and its associated component (tokenizer, Q-Former, and adapters) that processes all the historic visual tokens and the language instructions (navigation instruction and optional notice instruction), to predict the control signal and whether the given instruction is completed.
• Figure 5: The detailed structure of the vision encoder, which takes as input the multi-view multi-modality sensor data. In the pre-training stage, the vision encoder is appended with prediction headers to perform pre-training tasks (object detection, traffic light status classification, and future waypoint prediction). In the instruction-finetuning stage and inference stage, the prediction headers are discarded, and the vision encoder is frozen to generate visual tokens to feed into the LLM.