LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
Zhijie Qiao, Haowei Li, Zhong Cao, Henry X. Liu
TL;DR
LightEMMA proposes a lightweight, open-source end-to-end multimodal driving framework that leverages a broad set of vision-language models via zero-shot prompting. It systematically evaluates twelve VLMs on the nuScenes prediction task, measuring inference time, cost, reliability, and trajectory accuracy, and shows that larger models do not always yield better practical performance. The results highlight practical limitations of current VLMs for driving tasks and emphasize the value of modular design and task-oriented prompting in achieving robust, safe behavior. This work provides a benchmark framework to accelerate research and comparison in VLM-driven autonomous driving.
Abstract
Vision-Language Models (VLMs) have demonstrated significant potential for end-to-end autonomous driving. However, the field still lacks a practical platform that enables dynamic model updates, rapid validation, fair comparison, and intuitive performance assessment. To that end, we introduce LightEMMA, a Lightweight End-to-End Multimodal Model for Autonomous driving. LightEMMA provides a unified, VLM-based autonomous driving framework without ad hoc customizations, enabling easy integration with evolving state-of-the-art commercial and open-source models. We construct twelve autonomous driving agents using various VLMs and evaluate their performance on the challenging nuScenes prediction task, comprehensively assessing computational metrics and providing critical insights. Illustrative examples show that, although VLMs exhibit strong scenario interpretation capabilities, their practical performance in autonomous driving tasks remains a concern. Additionally, increased model complexity and extended reasoning do not necessarily lead to better performance, emphasizing the need for further improvements and task-specific designs. The code is available at https://github.com/michigan-traffic-lab/LightEMMA.