Large Language Model-assisted Autonomous Vehicle Recovery from Immobilization

TL;DR

The paper tackles AV immobilization in complex traffic where standard planning can fail, proposing StuckSolver, an LLM-driven add-on that detects immobilization from sensor data and generates high-level recovery commands executable by the native planner. Operating in zero-shot mode with Chain-of-Thought reasoning and optional passenger guidance via a Function-Calling API, StuckSolver can override decisions when needed while integrating without modifying core AV architecture. Experiments on Bench2Drive within CARLA show near-state-of-the-art driving scores, with further gains when passenger input is used, demonstrating improved resilience and accessibility in challenging scenarios. The work highlights a practical, scalable approach to enhance AV robustness through modular, interpretable high-level decision making that can adapt to human guidance and edge cases, with clear pathways for efficiency improvements and real-world deployment.

Abstract

Despite significant advancements in recent decades, autonomous vehicles (AVs) continue to face challenges in navigating certain traffic scenarios where human drivers excel. In such situations, AVs often become immobilized, disrupting overall traffic flow. Current recovery solutions, such as remote intervention (which is costly and inefficient) and manual takeover (which excludes non-drivers and limits AV accessibility), are inadequate. This paper introduces StuckSolver, a novel Large Language Model (LLM) driven recovery framework that enables AVs to resolve immobilization scenarios through self-reasoning and/or passenger-guided decision-making. StuckSolver is designed as a plug-in add-on module that operates on top of the AV's existing perception-planning-control stack, requiring no modification to its internal architecture. Instead, it interfaces with standard sensor data streams to detect immobilization states, interpret environmental context, and generate high-level recovery commands that can be executed by the AV's native planner. We evaluate StuckSolver on the Bench2Drive benchmark and in custom-designed uncertainty scenarios. Results show that StuckSolver achieves near-state-of-the-art performance through autonomous self-reasoning alone and exhibits further improvements when passenger guidance is incorporated.

Figures (4)

• Figure 1: Illustration of AV immobilization in traffic. The white AV, carrying non-driving passengers, travels in the right lane. When the leading blue vehicle becomes disabled and the left lane is obstructed by a paper bag, the AV’s re-planning module fails to generate an alternative trajectory due to the perceived obstruction. Although the bag is a benign and traversable object, the AV misclassifies it as a non-traversable obstacle, resulting in a complete stop and prolonged immobilization.
• Figure 2: An overview of the LLM-assisted AV recovery method.
• Figure 3: Chain of Thought for StuckSolver. S is an abbreviation for Step. {immobilized? reason} represents whether the AV is immobilized and the corresponding reason.
• Figure 4: Experimental results for three representative scenarios: pedestrian crossing, unexpected door opening, and cyclist following. The left panels display RGB images captured by the ego vehicle’s front-view camera, while the right panels illustrate the corresponding decision-making processes of the StuckSolver.