Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception

TL;DR

The paper addresses the challenge of robust autonomous vehicle perception in dynamic environments by integrating Large Language Models (LLMs) to provide contextual reasoning and language-based fusion of multi-sensor data. It proposes a three-component framework consisting of a Sensor Data Processing module, an LLM Integration Layer, and a Decision Support Module to transform raw sensor inputs into semantic insights and actionable vehicle commands. Empirical results on KITTI, nuScenes, and Carla/SUMO show improvements in occlusion handling (≈15% accuracy gain) and faster decision-making (≈30% reduced reaction time), along with favorable contextual understanding metrics and assessments of energy and computational load. This approach aims to create safer, more adaptive, and user-centric autonomous driving systems by extending perception beyond traditional data-driven methods and enabling continuous improvement through memory-enhanced, domain-tuned reasoning.

Abstract

Autonomous vehicles (AVs) rely on sophisticated perception systems to interpret their surroundings, a cornerstone for safe navigation and decision-making. The integration of Large Language Models (LLMs) into AV perception frameworks offers an innovative approach to address challenges in dynamic environments, sensor fusion, and contextual reasoning. This paper presents a novel framework for incorporating LLMs into AV perception, enabling advanced contextual understanding, seamless sensor integration, and enhanced decision support. Experimental results demonstrate that LLMs significantly improve the accuracy and reliability of AV perception systems, paving the way for safer and more intelligent autonomous driving technologies. By expanding the scope of perception beyond traditional methods, LLMs contribute to creating a more adaptive and human-centric driving ecosystem, making autonomous vehicles more reliable and transparent in their operations. These advancements redefine the relationship between human drivers and autonomous systems, fostering trust through enhanced understanding and personalized decision-making. Furthermore, by integrating memory modules and adaptive learning mechanisms, LLMs introduce continuous improvement in AV perception, enabling vehicles to evolve with time and adapt to changing environments and user preferences.