Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AuDeRe: Automated Strategy Decision and Realization in Robot Planning and Control via LLMs

Yue Meng, Fei Chen, Yongchao Chen, Chuchu Fan

TL;DR

AuDeRe addresses the challenge of flexible robotic planning and control by enabling automated strategy selection through large language models (LLMs) rather than relying on direct trajectory generation or a single fixed tool. The framework introduces an environment module and eight planning-control APIs, coordinated by an LLM that outputs integration code and iteratively refines decisions via feedback. Experiments across five increasingly complex scenarios show that the LLM-use-API approach yields higher success rates and requires fewer iterative queries than end-to-end trajectory or code generation baselines. The work demonstrates strong generalizability and potential to reduce manual engineering in diverse robotic tasks, with future work extending hardware validation and broader task coverage.

Abstract

Recent advancements in large language models (LLMs) have shown significant promise in various domains, especially robotics. However, most prior LLM-based work in robotic applications either directly predicts waypoints or applies LLMs within fixed tool integration frameworks, offering limited flexibility in exploring and configuring solutions best suited to different tasks. In this work, we propose a framework that leverages LLMs to select appropriate planning and control strategies based on task descriptions, environmental constraints, and system dynamics. These strategies are then executed by calling the available comprehensive planning and control APIs. Our approach employs iterative LLM-based reasoning with performance feedback to refine the algorithm selection. We validate our approach through extensive experiments across tasks of varying complexity, from simple tracking to complex planning scenarios involving spatiotemporal constraints. The results demonstrate that using LLMs to determine planning and control strategies from natural language descriptions significantly enhances robotic autonomy while reducing the need for extensive manual tuning and expert knowledge. Furthermore, our framework maintains generalizability across different tasks and notably outperforms baseline methods that rely on LLMs for direct trajectory, control sequence, or code generation.

AuDeRe: Automated Strategy Decision and Realization in Robot Planning and Control via LLMs

TL;DR

AuDeRe addresses the challenge of flexible robotic planning and control by enabling automated strategy selection through large language models (LLMs) rather than relying on direct trajectory generation or a single fixed tool. The framework introduces an environment module and eight planning-control APIs, coordinated by an LLM that outputs integration code and iteratively refines decisions via feedback. Experiments across five increasingly complex scenarios show that the LLM-use-API approach yields higher success rates and requires fewer iterative queries than end-to-end trajectory or code generation baselines. The work demonstrates strong generalizability and potential to reduce manual engineering in diverse robotic tasks, with future work extending hardware validation and broader task coverage.

Abstract

Recent advancements in large language models (LLMs) have shown significant promise in various domains, especially robotics. However, most prior LLM-based work in robotic applications either directly predicts waypoints or applies LLMs within fixed tool integration frameworks, offering limited flexibility in exploring and configuring solutions best suited to different tasks. In this work, we propose a framework that leverages LLMs to select appropriate planning and control strategies based on task descriptions, environmental constraints, and system dynamics. These strategies are then executed by calling the available comprehensive planning and control APIs. Our approach employs iterative LLM-based reasoning with performance feedback to refine the algorithm selection. We validate our approach through extensive experiments across tasks of varying complexity, from simple tracking to complex planning scenarios involving spatiotemporal constraints. The results demonstrate that using LLMs to determine planning and control strategies from natural language descriptions significantly enhances robotic autonomy while reducing the need for extensive manual tuning and expert knowledge. Furthermore, our framework maintains generalizability across different tasks and notably outperforms baseline methods that rely on LLMs for direct trajectory, control sequence, or code generation.

Paper Structure

This paper contains 16 sections, 1 equation, 10 figures.

Table of Contents

  1. Introduction
  2. Literature review
  3. Statement of contributions
  4. Problem statement
  5. LLM-based strategy decision
  6. Approach in a nutshell
  7. Environment module
  8. Planning-control API module
  9. LLM module and feedback loop
  10. Experiments
  11. Scenarios
  12. Baselines
  13. Performance metrics
  14. Experiment results
  15. Ablation study
  16. ...and 1 more sections

Figures (10)

  • Figure 1: LLM-use-API approach outperforms direct trajectory or code planners using LLMs.
  • Figure 2: The architecture for LLM-based strategy decision.
  • Figure 3: Example prompt for a simple tracking problem with a single iteration.
  • Figure 4: Robot planning and control scenarios with different complexities.
  • Figure 5: Comparison of success rates (left) and number of query rounds (right) across various LLM-based approaches with varying task complexities.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Remark 1