MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

TL;DR

MHRC addresses decentralized collaboration among mobile, manipulation, and mobile-manipulation robots using LLMs within a DEC-POMDP framework. It introduces Observation, Memory, and Planning modules and a description function to translate observations into prompts; employs chain-of-thought prompts and rich textual feedback to enable iterative replanning. Evaluations in PyBullet demonstrate high task success and efficient collaboration, with GPT-4o outperforming other LLMs and ablation studies confirming the importance of feedback and memory. The work advances decentralized, language-guided planning for heterogeneous robot teams and suggests future extensions to more robot types and more optimized CoT strategies.

Abstract

The integration of large language models (LLMs) with robotics has significantly advanced robots' abilities in perception, cognition, and task planning. The use of natural language interfaces offers a unified approach for expressing the capability differences of heterogeneous robots, facilitating communication between them, and enabling seamless task allocation and collaboration. Currently, the utilization of LLMs to achieve decentralized multi-heterogeneous robot collaborative tasks remains an under-explored area of research. In this paper, we introduce a novel framework that utilizes LLMs to achieve decentralized collaboration among multiple heterogeneous robots. Our framework supports three robot categories, mobile robots, manipulation robots, and mobile manipulation robots, working together to complete tasks such as exploration, transportation, and organization. We developed a rich set of textual feedback mechanisms and chain-of-thought (CoT) prompts to enhance task planning efficiency and overall system performance. The mobile manipulation robot can adjust its base position flexibly, ensuring optimal conditions for grasping tasks. The manipulation robot can comprehend task requirements, seek assistance when necessary, and handle objects appropriately. Meanwhile, the mobile robot can explore the environment extensively, map object locations, and communicate this information to the mobile manipulation robot, thus improving task execution efficiency. We evaluated the framework using PyBullet, creating scenarios with three different room layouts and three distinct operational tasks. We tested various LLM models and conducted ablation studies to assess the contributions of different modules. The experimental results confirm the effectiveness and necessity of our proposed framework.

Figures (3)

• Figure 1: This figure depicts the experimental scenarios and tasks involved in our research. The furniture types and layouts differ across various settings, with distinct room configurations, such as kitchens, bathrooms, and bedrooms. The task design is inspired by RoCo mandi2024roco, encompassing activities such as sandwich making, sorting solid objects, and packing items.
• Figure 2: The figure presents the overall workflow using a representative example. The "Start" and "Goal" denote the initial and target states of a task, respectively. Each robot autonomously makes decisions, plans, and executes atomic actions based on its local observations and communications received from other robots. The robots continuously replan their actions in response to environmental feedback. Through coordinated collaboration, multiple heterogeneous robots work together to accomplish complex, long-sequence tasks.
• Figure 3: The key prompts for the work are divided into two components: the system prompt and the user prompt. The system prompt(top) remains constant throughout the task, while the user prompt(bottom) dynamically evolves in response to the progression of the task. The figure uses a mobile manipulation robot as an example to illustrate prompt design. For prompt design of other robots and more detailed content, please refer to the appendix \ref{['appendix:prompt']}.