LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving

TL;DR

LimSim++ addresses the challenge of evaluating and advancing (M)LLMs in autonomous driving within long-term closed-loop simulations. It combines information from SUMO and CARLA into a unified, multimodal prompt-driven driver agent powered by (M)LLMs, augmented by a memory and reflection loop for continuous learning. The work introduces an open-source evaluation platform and a baseline closed-loop framework with memory modules, validated across intersections, roundabouts, and ramps, demonstrating memory-driven improvements in decision quality. This platform enables scalable, long-horizon testing and iterative improvement of knowledge-driven driving agents, supporting prompt engineering, model evaluation, and framework enhancement with practical impact for autonomous driving research.

Abstract

The emergence of Multimodal Large Language Models ((M)LLMs) has ushered in new avenues in artificial intelligence, particularly for autonomous driving by offering enhanced understanding and reasoning capabilities. This paper introduces LimSim++, an extended version of LimSim designed for the application of (M)LLMs in autonomous driving. Acknowledging the limitations of existing simulation platforms, LimSim++ addresses the need for a long-term closed-loop infrastructure supporting continuous learning and improved generalization in autonomous driving. The platform offers extended-duration, multi-scenario simulations, providing crucial information for (M)LLM-driven vehicles. Users can engage in prompt engineering, model evaluation, and framework enhancement, making LimSim++ a versatile tool for research and practice. This paper additionally introduces a baseline (M)LLM-driven framework, systematically validated through quantitative experiments across diverse scenarios. The open-source resources of LimSim++ are available at: https://pjlab-adg.github.io/limsim-plus/.

Figures (6)

• Figure 1: Platform composition. LimSim++ is the first closed-loop evaluation platform specifically developed for (M)LLM-driven autonomous driving.
• Figure 2: Framework of LimSim++. (1) Information Integration: Scenarios provided by SUMO and visual contents from CARLA are integrated into LimSim++ through the bridging module. (2) Prompt Engine: Constructing multimodal prompts to support (M)LLMs for understanding scenarios and tasks, including VLMs and LLMs. (3) Continuous Learning: The driver agent, driven by (M)LLM, makes behavioral decisions and continually enhances decision-making capabilities through mechanisms such as evaluation, reflection, memory, and tool library.
• Figure 3: Prompts and driving decisions for critical scenarios. RGB]189,218,165Green highlights the right answer from the GPT-4.
• Figure 4: Comparison of driver agents with and without memory in terms of efficiency, comfort, and safety. Notably, the agent with memory uses 10 memory items with 3 shots.
• Figure 5: Enhancement with memory in an intersection. RGB]189,218,165Green highlights the right answer. RGB]250,156,154Red highlights the wrong answer. RGB]255,204,102Yellow highlights the similar experience drawn from the memory module, which includes the past scenario descriptions and correct reasoning processes.