Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation

Jonathan Styrud, Matteo Iovino, Mikael Norrlöf, Mårten Björkman, Christian Smith

TL;DR

The paper tackles the challenge of quickly configuring robust and transparent manipulation policies in dynamic environments. It introduces BETR-XP-LLM, which couples large language models with a reactive task planner to dynamically expand Behavior Trees and resolve planning/execution failures, resulting in automatic policy updates. Key contributions include an improved prompt that eliminates reflective feedback, LLM-driven failure resolution for missing preconditions and parameters, and real-world ABB YuMi deployments demonstrating practical viability. The work offers a path toward autonomous, verifiable robotic manipulation with reduced reliance on manual intervention and extensive pre-programming, aided by open-source prompts and code.

Abstract

Robotic systems for manipulation tasks are increasingly expected to be easy to configure for new tasks or unpredictable environments, while keeping a transparent policy that is readable and verifiable by humans. We propose the method BEhavior TRee eXPansion with Large Language Models (BETR-XP-LLM) to dynamically and automatically expand and configure Behavior Trees as policies for robot control. The method utilizes an LLM to resolve errors outside the task planner's capabilities, both during planning and execution. We show that the method is able to solve a variety of tasks and failures and permanently update the policy to handle similar problems in the future.

Automatic Behavior Tree Expansion with LLMs for Robotic Manipulation

TL;DR

The paper tackles the challenge of quickly configuring robust and transparent manipulation policies in dynamic environments. It introduces BETR-XP-LLM, which couples large language models with a reactive task planner to dynamically expand Behavior Trees and resolve planning/execution failures, resulting in automatic policy updates. Key contributions include an improved prompt that eliminates reflective feedback, LLM-driven failure resolution for missing preconditions and parameters, and real-world ABB YuMi deployments demonstrating practical viability. The work offers a path toward autonomous, verifiable robotic manipulation with reduced reliance on manual intervention and extensive pre-programming, aided by open-source prompts and code.

Abstract

Robotic systems for manipulation tasks are increasingly expected to be easy to configure for new tasks or unpredictable environments, while keeping a transparent policy that is readable and verifiable by humans. We propose the method BEhavior TRee eXPansion with Large Language Models (BETR-XP-LLM) to dynamically and automatically expand and configure Behavior Trees as policies for robot control. The method utilizes an LLM to resolve errors outside the task planner's capabilities, both during planning and execution. We show that the method is able to solve a variety of tasks and failures and permanently update the policy to handle similar problems in the future.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. LLMs for robot program generation
  4. Behavior Trees
  5. Behavior Tree creation
  6. Method
  7. Experiments and results
  8. LLM-OBTEA comparisons
  9. Identifying missing preconditions
  10. Selecting missing parameters
  11. Real robot experiments
  12. Conclusions
  13. Future work

Figures (3)

  • Figure 1: ABB YuMi robot performing task 4: inserting a test tube into the centrifuge. The Kinect camera used for object detection can be seen mounted at the top.
  • Figure 2: Graphic representation showing all the components of BETR-XP-LLM. Green boxes denote algorithms and blue boxes denote data. Dashed lines and boxes denote optional components.
  • Figure 3: Example Behavior tree before (a) and after the failure resolution algorithm (b) for a cube pick and place task.