Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

Kehui Liu, Zixin Tang, Dong Wang, Zhigang Wang, Xuelong Li, Bin Zhao

TL;DR

COHERENT introduces a centralized hierarchical framework for planning with heterogeneous multi-robot systems by employing Proposal-Execution-Feedback-Adjustment (PEFA) cycles. A Task Assigner LLM decomposes high-level tasks into subtasks and assigns them to robot executors, who use Robot Executor LLMs to select feasible actions from robot-specific lists; execution feedback refines subsequent proposals. The approach is validated on a BEHAVIOR-1K-based benchmark with 5 scenes and 100 long-horizon tasks, where COHERENT outperforms baselines in success rate and efficiency, and ablations demonstrate the value of dialogue history. Real-world experiments with quadrotors, a robotic dog with an arm, and a robotics arm confirm zero-shot transfer and practical applicability. The work advances centralized planning for heterogeneous robotics and provides a challenging benchmark for future research.

Abstract

Leveraging the powerful reasoning capabilities of large language models (LLMs), recent LLM-based robot task planning methods yield promising results. However, they mainly focus on single or multiple homogeneous robots on simple tasks. Practically, complex long-horizon tasks always require collaboration among multiple heterogeneous robots especially with more complex action spaces, which makes these tasks more challenging. To this end, we propose COHERENT, a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems including quadrotors, robotic dogs, and robotic arms. Specifically, a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism is designed to decompose and assign actions for individual robots, where a centralized task assigner makes a task planning proposal to decompose the complex task into subtasks, and then assigns subtasks to robot executors. Each robot executor selects a feasible action to implement the assigned subtask and reports self-reflection feedback to the task assigner for plan adjustment. The PEFA loops until the task is completed. Moreover, we create a challenging heterogeneous multi-robot task planning benchmark encompassing 100 complex long-horizon tasks. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency. The experimental videos, code, and benchmark are released at https://github.com/MrKeee/COHERENT.

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

TL;DR

COHERENT introduces a centralized hierarchical framework for planning with heterogeneous multi-robot systems by employing Proposal-Execution-Feedback-Adjustment (PEFA) cycles. A Task Assigner LLM decomposes high-level tasks into subtasks and assigns them to robot executors, who use Robot Executor LLMs to select feasible actions from robot-specific lists; execution feedback refines subsequent proposals. The approach is validated on a BEHAVIOR-1K-based benchmark with 5 scenes and 100 long-horizon tasks, where COHERENT outperforms baselines in success rate and efficiency, and ablations demonstrate the value of dialogue history. Real-world experiments with quadrotors, a robotic dog with an arm, and a robotics arm confirm zero-shot transfer and practical applicability. The work advances centralized planning for heterogeneous robotics and provides a challenging benchmark for future research.

Abstract

Leveraging the powerful reasoning capabilities of large language models (LLMs), recent LLM-based robot task planning methods yield promising results. However, they mainly focus on single or multiple homogeneous robots on simple tasks. Practically, complex long-horizon tasks always require collaboration among multiple heterogeneous robots especially with more complex action spaces, which makes these tasks more challenging. To this end, we propose COHERENT, a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems including quadrotors, robotic dogs, and robotic arms. Specifically, a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism is designed to decompose and assign actions for individual robots, where a centralized task assigner makes a task planning proposal to decompose the complex task into subtasks, and then assigns subtasks to robot executors. Each robot executor selects a feasible action to implement the assigned subtask and reports self-reflection feedback to the task assigner for plan adjustment. The PEFA loops until the task is completed. Moreover, we create a challenging heterogeneous multi-robot task planning benchmark encompassing 100 complex long-horizon tasks. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency. The experimental videos, code, and benchmark are released at https://github.com/MrKeee/COHERENT.
Paper Structure (15 sections, 5 figures, 3 tables)

This paper contains 15 sections, 5 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Works
  3. Task Planning in Robotics
  4. Multi-robot Planners based on LLMs
  5. Method
  6. Task Planning Proposal
  7. Assigned Subtasks Execution
  8. Robot Execution Feedback
  9. Subtask Proposal Adjustment
  10. Experiments
  11. Benchmark
  12. Evaluation Metrics
  13. Simulation Experiments
  14. Real-world Experiment
  15. CONCLUSIONS

Figures (5)

  • Figure 1: Comparisons of LLM-based planning among (a) single-robot, (b) homogeneous multi-robot, and (c) heterogeneous multi-robot. In contrast to previous methods, COHERENT focuses on task allocation and collaboration among different types of robots, aiming to collaboratively accomplish complex and long-horizon tasks.
  • Figure 2: PEFA mechanism in COHERENT for heterogeneous multi-robot task planning. It consists of two stages, i.e., task assigner and robot executor. The proposal module decomposes the human instruction into subgoals that can be assigned to a specific robot. The execution module further maps the subgoal to an executable action. Additionally, the Feedback and Adjustment modules are designed for amendment or advancement of subtasks to complete the instruction.
  • Figure 3: The illustration of five scenes and three heterogeneous robots in our benchmark. In subfigure (c), the distribution of various types of tasks within the benchmark is displayed in the pie chart, along with the histogram showing the average ground truth (GT) steps for each type of task.
  • Figure 4: A successful example of putting the meat on the grill in scenario S5.
  • Figure 5: Real-world Experiment: With the cooperation of the robotic dog and robotic arm, the quadrotor swarm fly to the kitchen across a closed door and take the cookies and porridge back which are on the kitchen table initially. This task cannot be completed without the presence of any one of the robots.