Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bootstrapped LLM Semantics for Context-Aware Path Planning

Mani Amani, Behrad Beheshti, Reza Akhavian

TL;DR

The paper addresses semantic-aware path planning in human-centric environments by treating a large language model (LLM) as a stochastic semantic sensor. It uses Bayesian bootstrap to form a nonparametric posterior over per-class semantic risk and maps tail risk through a CVaR-based semantic potential to a classical DSP via a distance-plus-repulsion cost, enabling prompt-conditioned planning. A multi-heuristic A* framework with a consistent Euclidean anchor and an informative auxiliary heuristic maintains optimality while accelerating search. Across simulations, a BIM-backed digital twin, and real-world experiments, the approach yields safer, more context-aware trajectories and demonstrates the ability to adapt conservatism to explicit and implicit prompts without retraining.

Abstract

Prompting robots with natural language (NL) has largely been studied as what task to execute (goal selection, skill sequencing) rather than how to execute that task safely and efficiently in semantically rich, human-centric spaces. We address this gap with a framework that turns a large language model (LLM) into a stochastic semantic sensor whose outputs modulate a classical planner. Given a prompt and a semantic map, we draw multiple LLM "danger" judgments and apply a Bayesian bootstrap to approximate a posterior over per-class risk. Using statistics from the posterior, we create a potential cost to formulate a path planning problem. Across simulated environments and a BIM-backed digital twin, our method adapts how the robot moves in response to explicit prompts and implicit contextual information. We present qualitative and quantitative results.

Bootstrapped LLM Semantics for Context-Aware Path Planning

TL;DR

The paper addresses semantic-aware path planning in human-centric environments by treating a large language model (LLM) as a stochastic semantic sensor. It uses Bayesian bootstrap to form a nonparametric posterior over per-class semantic risk and maps tail risk through a CVaR-based semantic potential to a classical DSP via a distance-plus-repulsion cost, enabling prompt-conditioned planning. A multi-heuristic A* framework with a consistent Euclidean anchor and an informative auxiliary heuristic maintains optimality while accelerating search. Across simulations, a BIM-backed digital twin, and real-world experiments, the approach yields safer, more context-aware trajectories and demonstrates the ability to adapt conservatism to explicit and implicit prompts without retraining.

Abstract

Prompting robots with natural language (NL) has largely been studied as what task to execute (goal selection, skill sequencing) rather than how to execute that task safely and efficiently in semantically rich, human-centric spaces. We address this gap with a framework that turns a large language model (LLM) into a stochastic semantic sensor whose outputs modulate a classical planner. Given a prompt and a semantic map, we draw multiple LLM "danger" judgments and apply a Bayesian bootstrap to approximate a posterior over per-class risk. Using statistics from the posterior, we create a potential cost to formulate a path planning problem. Across simulated environments and a BIM-backed digital twin, our method adapts how the robot moves in response to explicit prompts and implicit contextual information. We present qualitative and quantitative results.

Paper Structure

This paper contains 14 sections, 1 theorem, 10 equations, 15 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Mathematical Framework
  4. Repulsive Fields
  5. Semantic Augmentation of Repulsive Fields
  6. Multi-Heuristic Path Planning
  7. Auxiliary heuristic (informative, may be inadmissible).
  8. Experiments and Results
  9. Scenario 1, explicit object avoidance
  10. Scenario 2 implicit object avoidance.
  11. Real-world evaluation through environment digital twin
  12. Ablation study
  13. Discussions and Limitations
  14. Conclusion

Key Result

Lemma 3.2

Assume $d(s,s') \ge \|x_s - x_{s'}\|_2$ for all neighbors (e.g., 4/8-connected grids with costs $1,\sqrt{2}$). Then the Euclidean distance $h_0(s)=\|x_s-x_g\|_2$ is consistent (hence admissible) for the combined cost $c(s,s')=d(s,s')+\gamma\,\Phi(s')$.

Figures (15)

  • Figure 1: Overall framework to support the context-aware understanding of the pathplanning problem
  • Figure 2: Overlay of paths given the prompt implying that the workzone is very busy so a more risk-aware path should be taken.
  • Figure 3: Overlay of paths given a prompt implying the work zone is very empty, so a more risk-neutral path should be taken.
  • Figure 4: Overlay of paths given the prompt implying that the forklift is empty and a path in that direction should be taken. Note that the potential field around the forklift is dimmer than the rest of the obstacles. This shows the efficacy of the framework in updating semantic costs depending on obstacle and prompt qualities.
  • Figure 5: Prompt notifying the robot that the plumbing team is in, implicitly implying that the plumbing storage unit might be active
  • ...and 10 more figures

Theorems & Definitions (3)

  • Definition 3.1: Consistent heuristic
  • Lemma 3.2: Euclidean anchor is consistent for $c$
  • proof