Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances

TL;DR

This survey addresses the integration of Large Language Models into autonomous driving by focusing on multi-agent systems that overcome the perception, collaboration, and compute limits of single-agent ADSs. It develops a taxonomy around agent environments/profiles, inter-agent interaction modes, and agent-human collaboration, and reviews three primary interaction types: multi-vehicle, vehicle-infrastructure, and vehicle-assistant, along with applications, datasets, and benchmarks. The work highlights mechanisms for collaborative perception and decision-making, as well as cloud-edge deployment and assistant-tools, while outlining key challenges such as safety, privacy, and real-world deployment. By mapping existing methods and resources, the paper aims to guide NLP researchers and the autonomous driving community toward scalable, safe, and human-aligned LLM-based multi-agent ADSs.

Abstract

Autonomous Driving Systems (ADSs) are revolutionizing transportation by reducing human intervention, improving operational efficiency, and enhancing safety. Large Language Models (LLMs) have been integrated into ADSs to support high-level decision-making through their powerful reasoning, instruction-following, and communication abilities. However, LLM-based single-agent ADSs face three major challenges: limited perception, insufficient collaboration, and high computational demands. To address these issues, recent advances in LLM-based multi-agent ADSs leverage language-driven communication and coordination to enhance inter-agent collaboration. This paper provides a frontier survey of this emerging intersection between NLP and multi-agent ADSs. We begin with a background introduction to related concepts, followed by a categorization of existing LLM-based methods based on different agent interaction modes. We then discuss agent-human interactions in scenarios where LLM-based agents engage with humans. Finally, we summarize key applications, datasets, and challenges to support future research.

Figures (7)

• Figure 1: Limitations of LLM-based single-agent ADSs. At an intersection without traffic lights, an accident has occurred ahead, causing Veh1 to be stuck. Due to limited perception, Veh1 is unable to assess the situation and cannot proceed. Veh2 intends to go straight, and Veh3 wants to turn left. However, due to insufficient collaboration, they are also unable to navigate the intersection efficiently. Furthermore, due to high computing demands, the lightweight agent on Veh1 struggles to handle the complex driving scenario and has to rely on a more powerful cloud-based agent for assistance.
• Figure 2: Overview of LLM-based (a) single- and (b) multi-agent ADSs, with key terms and differences highlighted.
• Figure 3: Different interaction modes and structures.
• Figure 4: A taxonomy of LLM-based Multi-Agent Autonomous Driving Systems.
• Figure 5: Two modes of agent-human interaction.