Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"Don't Do That!": Guiding Embodied Systems through Large Language Model-based Constraint Generation

Amin Seffo, Aladin Djuhera, Masataro Asai, Holger Boche

TL;DR

STPR is proposed, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions, thus circumventing complex reasoning and avoiding potential hallucinations.

Abstract

Recent advancements in large language models (LLMs) have spurred interest in robotic navigation that incorporates complex spatial, mathematical, and conditional constraints from natural language into the planning problem. Such constraints can be informal yet highly complex, making it challenging to translate into a formal description that can be passed on to a planning algorithm. In this paper, we propose STPR, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions. STPR leverages the LLM's strong coding capabilities to shift the problem description from language into structured and transparent code, thus circumventing complex reasoning and avoiding potential hallucinations. We show that these LLM-generated functions accurately describe even complex mathematical constraints, and apply them to point cloud representations with traditional search algorithms. Experiments in a simulated Gazebo environment show that STPR ensures full compliance across several constraints and scenarios, while having short runtimes. We also verify that STPR can be used with smaller, code-specific LLMs, making it applicable to a wide range of compact models at low inference cost.

"Don't Do That!": Guiding Embodied Systems through Large Language Model-based Constraint Generation

TL;DR

STPR is proposed, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions, thus circumventing complex reasoning and avoiding potential hallucinations.

Abstract

Recent advancements in large language models (LLMs) have spurred interest in robotic navigation that incorporates complex spatial, mathematical, and conditional constraints from natural language into the planning problem. Such constraints can be informal yet highly complex, making it challenging to translate into a formal description that can be passed on to a planning algorithm. In this paper, we propose STPR, a constraint generation framework that uses LLMs to translate constraints (expressed as instructions on ``what not to do'') into executable Python functions. STPR leverages the LLM's strong coding capabilities to shift the problem description from language into structured and transparent code, thus circumventing complex reasoning and avoiding potential hallucinations. We show that these LLM-generated functions accurately describe even complex mathematical constraints, and apply them to point cloud representations with traditional search algorithms. Experiments in a simulated Gazebo environment show that STPR ensures full compliance across several constraints and scenarios, while having short runtimes. We also verify that STPR can be used with smaller, code-specific LLMs, making it applicable to a wide range of compact models at low inference cost.

Paper Structure

This paper contains 18 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Proposed STPR Method
  4. LLM-Based Constraint Code Prompting
  5. Point-Cloud Sampling
  6. Constrained Path Planning
  7. Empirical Evaluations
  8. Simulation Scenarios
  9. Qualitative Results
  10. 1) Vanilla A$^*$/RRT*
  11. 2) STPR
  12. 3) Naive VLM Planning
  13. Quantitative Results
  14. 1) Runtime and Quality
  15. 2) Model Variations
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Gazebo environment...
  • Figure 2: STPR Overview: LLM generates a Python function based on user constraints using a prompt template. The function is then integrated into a point sampling pipeline to generate a constrained representation of the environment. A classical algorithm uses this point cloud for path planning.
  • Figure 3: Top: Prompt template for constraint code generation, including the System Instruction ("You are a robot ...") in orange, the Environment Block in black, the Constraint Block with user-specific instructions (here: Scenario 4: Fireplace Heat Avoidance) in blue, and the Python function signature. Bottom: Corresponding constraint function generated by the LLM.
  • Figure 4: Planning results for STPR and baselines. Blue: Point cloud for static objects. Red: Point cloud from constraint functions. Magenta: Path generated by vanilla A$^*$. Yellow: Path generated by vanilla RRT*. Black: Path generated by STPR-A*. Green: Path generated by STPR-RRT*. Cyan: Path generated by GPT-4o (using annotated image). Only STPR is compliant across all scenarios, refusing paths for (S1) and (S3), and avoiding hazards for (S2) and (S4).