SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving

TL;DR

This work tackles end-to-end autonomous driving by reframing perception, prediction, planning, and behavior as a Graph Visual Question Answering (GVQA) problem that is solved with a vision-language model. The architecture couples a vision encoder (InternViT-6B) with a language model (Vicuna-13B) through a GVQA graph, enabling stage-wise reasoning across four driving stages. Key contributions include exploiting GVQA logical dependencies, refining prompts, and introducing object-detection cues to enrich context, achieving a DriveLM-nuScenes score of 52.7 and superior language-aware accuracy over baselines. The approach demonstrates the potential of language-grounded, multimodal decision making for robust and interpretable end-to-end autonomous driving, with implications for safer and more adaptable driving systems.

Abstract

Many fields could benefit from the rapid development of the large language models (LLMs). The end-to-end autonomous driving (e2eAD) is one of the typically fields facing new opportunities as the LLMs have supported more and more modalities. Here, by utilizing vision-language model (VLM), we proposed an e2eAD method called SimpleLLM4AD. In our method, the e2eAD task are divided into four stages, which are perception, prediction, planning, and behavior. Each stage consists of several visual question answering (VQA) pairs and VQA pairs interconnect with each other constructing a graph called Graph VQA (GVQA). By reasoning each VQA pair in the GVQA through VLM stage by stage, our method could achieve e2e driving with language. In our method, vision transformers (ViT) models are employed to process nuScenes visual data, while VLM are utilized to interpret and reason about the information extracted from the visual inputs. In the perception stage, the system identifies and classifies objects from the driving environment. The prediction stage involves forecasting the potential movements of these objects. The planning stage utilizes the gathered information to develop a driving strategy, ensuring the safety and efficiency of the autonomous vehicle. Finally, the behavior stage translates the planned actions into executable commands for the vehicle. Our experiments demonstrate that SimpleLLM4AD achieves competitive performance in complex driving scenarios.