Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving

Xinmeng Hou, Wuqi Wang, Long Yang, Hao Lin, Jinglun Feng, Haigen Min, Xiangmo Zhao

TL;DR

DriveAgent tackles the challenge of robust, interpretable autonomous driving by integrating LLM-driven reasoning with multimodal sensor fusion across camera, LiDAR, GPS, and IMU. It adopts a four-module, multi-agent architecture that performs descriptive analysis, vehicle and environmental reasoning, and urgency-aware response generation. The approach achieves superior object recognition and multi-sensor reasoning accuracy on a real-world Chang’an University dataset, outperforming strong prompt-based baselines. The work highlights the value of modular, interpretable AI in driving and provides datasets, benchmarks, and fine-tuning strategies to advance sensor-aware, explainable autonomy.

Abstract

We introduce DriveAgent, a novel multi-agent autonomous driving framework that leverages large language model (LLM) reasoning combined with multimodal sensor fusion to enhance situational understanding and decision-making. DriveAgent uniquely integrates diverse sensor modalities-including camera, LiDAR, GPS, and IMU-with LLM-driven analytical processes structured across specialized agents. The framework operates through a modular agent-based pipeline comprising four principal modules: (i) a descriptive analysis agent identifying critical sensor data events based on filtered timestamps, (ii) dedicated vehicle-level analysis conducted by LiDAR and vision agents that collaboratively assess vehicle conditions and movements, (iii) environmental reasoning and causal analysis agents explaining contextual changes and their underlying mechanisms, and (iv) an urgency-aware decision-generation agent prioritizing insights and proposing timely maneuvers. This modular design empowers the LLM to effectively coordinate specialized perception and reasoning agents, delivering cohesive, interpretable insights into complex autonomous driving scenarios. Extensive experiments on challenging autonomous driving datasets demonstrate that DriveAgent is achieving superior performance on multiple metrics against baseline methods. These results validate the efficacy of the proposed LLM-driven multi-agent sensor fusion framework, underscoring its potential to substantially enhance the robustness and reliability of autonomous driving systems.

DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving

TL;DR

DriveAgent tackles the challenge of robust, interpretable autonomous driving by integrating LLM-driven reasoning with multimodal sensor fusion across camera, LiDAR, GPS, and IMU. It adopts a four-module, multi-agent architecture that performs descriptive analysis, vehicle and environmental reasoning, and urgency-aware response generation. The approach achieves superior object recognition and multi-sensor reasoning accuracy on a real-world Chang’an University dataset, outperforming strong prompt-based baselines. The work highlights the value of modular, interpretable AI in driving and provides datasets, benchmarks, and fine-tuning strategies to advance sensor-aware, explainable autonomy.

Abstract

We introduce DriveAgent, a novel multi-agent autonomous driving framework that leverages large language model (LLM) reasoning combined with multimodal sensor fusion to enhance situational understanding and decision-making. DriveAgent uniquely integrates diverse sensor modalities-including camera, LiDAR, GPS, and IMU-with LLM-driven analytical processes structured across specialized agents. The framework operates through a modular agent-based pipeline comprising four principal modules: (i) a descriptive analysis agent identifying critical sensor data events based on filtered timestamps, (ii) dedicated vehicle-level analysis conducted by LiDAR and vision agents that collaboratively assess vehicle conditions and movements, (iii) environmental reasoning and causal analysis agents explaining contextual changes and their underlying mechanisms, and (iv) an urgency-aware decision-generation agent prioritizing insights and proposing timely maneuvers. This modular design empowers the LLM to effectively coordinate specialized perception and reasoning agents, delivering cohesive, interpretable insights into complex autonomous driving scenarios. Extensive experiments on challenging autonomous driving datasets demonstrate that DriveAgent is achieving superior performance on multiple metrics against baseline methods. These results validate the efficacy of the proposed LLM-driven multi-agent sensor fusion framework, underscoring its potential to substantially enhance the robustness and reliability of autonomous driving systems.
Paper Structure (20 sections, 9 equations, 5 figures, 4 tables, 2 algorithms)

This paper contains 20 sections, 9 equations, 5 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Module 1: Descriptive Analysis
  4. Module 2: Vehicle Reasoning
  5. Vision Descriptor
  6. LiDAR Descriptor
  7. Vehicle Status Reasoning
  8. Module 3: Environmental Reasoning
  9. Environmental Reasoning
  10. Causal Analysis
  11. Module 4: Response Generation
  12. Experiments
  13. Datasets
  14. Task and Evaluation Metrics
  15. Baseline Approaches
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of the inputs and outputs for the proposed DriveAgent framework. DriveAgent takes multimodal sensor data as inputs, including camera, Lidar GPS, and IMU data. The input data are processed through four structured tasks (T1 to T4), supporting comprehensive reasoning tasks at both the environment level and the vehicle level as outputs.
  • Figure 2: An overview of the proposed architecture which is consisting of four modules (M1 to M4), where multimodal sensor inputs—camera, IMU, GPS and LiDAR—enable both environment-level tasks (e.g., information retrieval, environmental change detection, and reasoning) and vehicle-level tasks (e.g., vehicle status analysis, motion evaluation, and behavior pattern recognition).
  • Figure 3: Data collection vehicle sensor configuration and satellite images of recorded driving routes. There are three routes being recorded in total at Chang'an University, Xi'an, China. Route 1 (R1) is shown in red trajectory, Route 2 (R2) is shown in purple trajectory, while Route 3 (R3) is shown in green trajectory.
  • Figure 4: Overview of the multimodal reasoning pipeline used for driving scene understanding. Visual descriptions are generated from camera images, focusing on identifying traffic-related objects and maintaining objective scene summaries. LiDAR-based descriptions analyze object sizes and relative positions to assess driving risk. In the reasoning stages, LLM agents evaluate the correctness of sensor-based analyses (vehicle reasoning) and identify environmental changes over time (environmental reasoning). Human instructions and corresponding LLM generations are provided for each step, supporting robust, explainable autonomous driving assessments.
  • Figure 5: Distribution of object categories in the human-annotated ground truth versus each model’s predictions. Colour key: deep blue = fixed installations, orange = four-wheel vehicles, grey = non-four-wheel vehicles, yellow = plants, and light blue = monitors.