Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Behavior Tree Generation using Large Language Models for Sequential Manipulation Planning with Human Instructions and Feedback

Jicong Ao, Yansong Wu, Fan Wu, Sami Haddadin

TL;DR

This is the first human-guided LLM-based BT generation framework that unifies various plausible ways of using LLMs to fully generate BTs that are executable on the real testbed and take into account granular knowledge of tool use.

Abstract

In this work, we propose an LLM-based BT generation framework to leverage the strengths of both for sequential manipulation planning. To enable human-robot collaborative task planning and enhance intuitive robot programming by nonexperts, the framework takes human instructions to initiate the generation of action sequences and human feedback to refine BT generation in runtime. All presented methods within the framework are tested on a real robotic assembly example, which uses a gear set model from the Siemens Robot Assembly Challenge. We use a single manipulator with a tool-changing mechanism, a common practice in flexible manufacturing, to facilitate robust grasping of a large variety of objects. Experimental results are evaluated regarding success rate, logical coherence, executability, time consumption, and token consumption. To our knowledge, this is the first human-guided LLM-based BT generation framework that unifies various plausible ways of using LLMs to fully generate BTs that are executable on the real testbed and take into account granular knowledge of tool use.

Behavior Tree Generation using Large Language Models for Sequential Manipulation Planning with Human Instructions and Feedback

TL;DR

This is the first human-guided LLM-based BT generation framework that unifies various plausible ways of using LLMs to fully generate BTs that are executable on the real testbed and take into account granular knowledge of tool use.

Abstract

In this work, we propose an LLM-based BT generation framework to leverage the strengths of both for sequential manipulation planning. To enable human-robot collaborative task planning and enhance intuitive robot programming by nonexperts, the framework takes human instructions to initiate the generation of action sequences and human feedback to refine BT generation in runtime. All presented methods within the framework are tested on a real robotic assembly example, which uses a gear set model from the Siemens Robot Assembly Challenge. We use a single manipulator with a tool-changing mechanism, a common practice in flexible manufacturing, to facilitate robust grasping of a large variety of objects. Experimental results are evaluated regarding success rate, logical coherence, executability, time consumption, and token consumption. To our knowledge, this is the first human-guided LLM-based BT generation framework that unifies various plausible ways of using LLMs to fully generate BTs that are executable on the real testbed and take into account granular knowledge of tool use.
Paper Structure (10 sections, 4 figures, 1 table, 1 algorithm)

This paper contains 10 sections, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Framework
  3. LLM-ENHANCED ROBOT TASK PLANNING AND EXECUTION FRAMEWORK
  4. LLM-ENHANCED BEHAVIOR TREE GENERATION METHODS
  5. Results
  6. Experiment Setup
  7. Evaluation Metrics
  8. Exemplar Behavior Tree
  9. Behavior Tree Expansion Algorithm
  10. Example of Behavior Tree Generation and Execution

Figures (4)

  • Figure 1: The basic workflow and the four proposed methods, where the red dashed rectangles show the place in the workflow that the contents from different methods can substitute.
  • Figure 2: Experiment setup with a franka panda robot, four toolcubes from Leverage, and a gearset from the Siemens Robot Assembly Challenge
  • Figure 3: An exemplar behavior tree generated from the assembly step "insert the shaft 1 into the gearbase hole 1".
  • Figure 4: Robotic assembly of a gear set. Shown are the generated behavior tree and the corresponding sequence of actions. The order of actions is labeled by number and shown from left to right, while their corresponding action nodes in the BT are colored green.