LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning

TL;DR

This work tackles the planning gap in autonomous driving by introducing LLM-Assist, a hybrid framework that augments a strong rule-based base planner with Large Language Model reasoning. The approach uses PDM-Closed as the base planner to generate multiple trajectory proposals and scores them, invoking an LLM-based planner only when safety or comfort metrics fall below thresholds, with two modes: unconstrained trajectory generation and parameterized guidance for the base planner. The parameterized variant (LLM-Assist_par) yields substantial improvements, achieving state-of-the-art results on the nuPlan benchmark across closed-loop challenges and reducing dangerous driving events by about 11% versus the prior best. The results demonstrate the value of grounding LLM reasoning in a solid planner while highlighting limitations such as perception grounding, latency, and potential hallucinations, and point to future work in multimodal grounding and faster, more reliable LLM integration for real-time autonomous navigation.

Abstract

Although planning is a crucial component of the autonomous driving stack, researchers have yet to develop robust planning algorithms that are capable of safely handling the diverse range of possible driving scenarios. Learning-based planners suffer from overfitting and poor long-tail performance. On the other hand, rule-based planners generalize well, but might fail to handle scenarios that require complex driving maneuvers. To address these limitations, we investigate the possibility of leveraging the common-sense reasoning capabilities of Large Language Models (LLMs) such as GPT4 and Llama2 to generate plans for self-driving vehicles. In particular, we develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Guided by commonsense reasoning abilities of LLMs, our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach. Through extensive evaluation on the nuPlan benchmark, we achieve state-of-the-art performance, outperforming all existing pure learning- and rule-based methods across most metrics. Our code will be available at https://llmassist.github.io.

Figures (10)

• Figure 1: Architecture of LLM-Assist. We propose a novel hybrid planning approach that leverages a SoTA rule-based planner, PDM-Closed, for common scenarios and a novel LLM-based planner, for challenging high uncertainty scenarios. When the PDM-generated scores are deemed insufficient—falling below predetermined thresholds for various metrics such as collision risk and passenger comfort—we invoke the LLM-Planner.
• Figure 2: System Prompt for $\textsc{LLM-Assist}_{\textsc{unc}}$.
• Figure 3: System Prompt for $\textsc{LLM-Assist}_{\textsc{par}}$.
• Figure 4: ROC Curve of PDM-Closed's Predicted Proposal Scores. Evaluated on nuPlan Closed-Loop Challenges Val14 split.
• Figure 5: Qualitative Comparisons between LLM-Assist and PDMClosed.