Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation

Achref Doula, Max Mühlhäuser, Alejandro Sanchez Guinea

TL;DR

The paper tackles the safety problem of using LLMs for autonomous navigation under uncertainty. It proposes SafePath, a three-stage framework combining LLM-based path generation with conformal prediction to guarantee safety with a user-defined probability, plus a decision rule that balances autonomy and human delegation. Empirical results on nuScenes and Highway-env show substantial reductions in planning uncertainty and collision rates, and demonstrate the value of formal safety guarantees in LLM-driven planning. The work offers theoretical guarantees and practical evidence that conformal prediction can enable safer deployment of LLMs in safety-critical autonomous systems.

Abstract

Large Language Models (LLMs) show growing promise in autonomous driving by reasoning over complex traffic scenarios to generate path plans. However, their tendencies toward overconfidence, and hallucinations raise critical safety concerns. We introduce SafePath, a modular framework that augments LLM-based path planning with formal safety guarantees using conformal prediction. SafePath operates in three stages. In the first stage, we use an LLM that generates a set of diverse candidate paths, exploring possible trajectories based on agent behaviors and environmental cues. In the second stage, SafePath filters out high-risk trajectories while guaranteeing that at least one safe option is included with a user-defined probability, through a multiple-choice question-answering formulation that integrates conformal prediction. In the final stage, our approach selects the path with the lowest expected collision risk when uncertainty is low or delegates control to a human when uncertainty is high. We theoretically prove that SafePath guarantees a safe trajectory with a user-defined probability, and we show how its human delegation rate can be tuned to balance autonomy and safety. Extensive experiments on nuScenes and Highway-env show that SafePath reduces planning uncertainty by 77\% and collision rates by up to 70\%, demonstrating effectiveness in making LLM-driven path planning more safer.

SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation

TL;DR

The paper tackles the safety problem of using LLMs for autonomous navigation under uncertainty. It proposes SafePath, a three-stage framework combining LLM-based path generation with conformal prediction to guarantee safety with a user-defined probability, plus a decision rule that balances autonomy and human delegation. Empirical results on nuScenes and Highway-env show substantial reductions in planning uncertainty and collision rates, and demonstrate the value of formal safety guarantees in LLM-driven planning. The work offers theoretical guarantees and practical evidence that conformal prediction can enable safer deployment of LLMs in safety-critical autonomous systems.

Abstract

Large Language Models (LLMs) show growing promise in autonomous driving by reasoning over complex traffic scenarios to generate path plans. However, their tendencies toward overconfidence, and hallucinations raise critical safety concerns. We introduce SafePath, a modular framework that augments LLM-based path planning with formal safety guarantees using conformal prediction. SafePath operates in three stages. In the first stage, we use an LLM that generates a set of diverse candidate paths, exploring possible trajectories based on agent behaviors and environmental cues. In the second stage, SafePath filters out high-risk trajectories while guaranteeing that at least one safe option is included with a user-defined probability, through a multiple-choice question-answering formulation that integrates conformal prediction. In the final stage, our approach selects the path with the lowest expected collision risk when uncertainty is low or delegates control to a human when uncertainty is high. We theoretically prove that SafePath guarantees a safe trajectory with a user-defined probability, and we show how its human delegation rate can be tuned to balance autonomy and safety. Extensive experiments on nuScenes and Highway-env show that SafePath reduces planning uncertainty by 77\% and collision rates by up to 70\%, demonstrating effectiveness in making LLM-driven path planning more safer.
Paper Structure (31 sections, 3 theorems, 14 equations, 8 figures, 8 tables, 1 algorithm)

This paper contains 31 sections, 3 theorems, 14 equations, 8 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Background on Conformal Prediction
  4. Approach
  5. Approach Overview
  6. Path Generation
  7. Uncertainty-Aware Path Selection
  8. Path Decision
  9. Implications.
  10. Evaluation
  11. Dataset and Simulation Environments
  12. Implementation
  13. Effects of Path Similarity Threshold $\delta$
  14. Conformal Prediction Evaluation
  15. Baselines.
  16. ...and 16 more sections

Key Result

Theorem 3.1

Let $D_{\text{cal}} = \{(x_i, y_i)\}_{i=1}^n$ be a calibration set of exchangeable samples, and let $S$ be a chosen non-conformity score. For each $(x_i, y_i)\in D_{\text{cal}}$, compute $S(x_i, y_i)$ and let $\hat{q}$ be the $(1-\alpha)$-quantile of these scores. Then for a new input $x_{\text{test satisfies $\mathbb{P}\bigl[y_{\text{test}} \in C(x_{\text{test}})\bigr] \,\ge\, 1 - \alpha.$

Figures (8)

  • Figure 1: Overview of SafePath for autonomous vehicle path planning with LLMs. The approach consists of three stages: (1) Path Generation, where $\text{LLM}_1$ generates a set of candidate paths $P$ based on the autonomous vehicle's goals and environmental state derived from a textual scene description; (2) Uncertainty-Aware Path Selection, where $\text{LLM}_2$ refines the paths in $P$ by framing the problem as a multiple-choice question-answering task and performing a conformal prediction step to construct a prediction set $C$ that includes a safe path with user-defined probability $1-\alpha$; and (3) Path Decision, which evaluates the selected path(s) based on uncertainty levels to determine if the vehicle can proceed autonomously by selecting the path with the least expected collision risk or if human intervention is required due to high uncertainty. The prompts shown in the Figure (Task Prompt, Multiple-Choice Question Formulation Prompt) are partial examples for illustration. The complete prompts are provided in Appendix \ref{['app:prompt']}.
  • Figure 2: Human delegation rate as a function of the similarity threshold $\delta$ for different $\alpha$ values.
  • Figure 3: Results of the closed-loop simulation on the Highway-env comparing GPT-Driver to SafePath.
  • Figure 4: Path generation through several successive samples for different situations.
  • Figure 5: Predicted paths and path decisions for different situations.
  • ...and 3 more figures

Theorems & Definitions (7)

  • Theorem 3.1: Conformal Prediction
  • Definition 4.1: Safe Path
  • Theorem 4.1: Prediction Set Safety Guarantee
  • Definition 4.2: $\delta$-Equivalence
  • Theorem 4.2: SafePath Guarantee
  • Proof A.1
  • Proof A.2