Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models

Zhaoxing Li, Wenbo Wu, Yue Wang, Yanran Xu, William Hunt, Sebastian Stein

TL;DR

The paper tackles generalization, heterogeneity, and safety challenges in multi-robot systems by introducing HMCF, a human-in-the-loop framework that leverages LLM-based agents for per-robot reasoning and a central assistant for task allocation, augmented with human oversight to mitigate hallucinations. It integrates retrieval-augmented generation for easy onboarding of new robots and provides a user-friendly web interface for oversight, enabling robust, zero-shot generalization across diverse tasks and environments. Empirical results in BEHAVIOR-1K show a 4.76% improvement in task success rate with HMCF, and real-world deployment confirms adaptive, multi-robot coordination with minimal human intervention. While effective, the approach faces scalability and connectivity challenges for large fleets and cloud-dependent LLMs, guiding future work on scalable communication and user-centric refinements.

Abstract

Rapid advancements in artificial intelligence (AI) have enabled robots to performcomplex tasks autonomously with increasing precision. However, multi-robot systems (MRSs) face challenges in generalization, heterogeneity, and safety, especially when scaling to large-scale deployments like disaster response. Traditional approaches often lack generalization, requiring extensive engineering for new tasks and scenarios, and struggle with managing diverse robots. To overcome these limitations, we propose a Human-in-the-loop Multi-Robot Collaboration Framework (HMCF) powered by large language models (LLMs). LLMs enhance adaptability by reasoning over diverse tasks and robot capabilities, while human oversight ensures safety and reliability, intervening only when necessary. Our framework seamlessly integrates human oversight, LLM agents, and heterogeneous robots to optimize task allocation and execution. Each robot is equipped with an LLM agent capable of understanding its capabilities, converting tasks into executable instructions, and reducing hallucinations through task verification and human supervision. Simulation results show that our framework outperforms state-of-the-art task planning methods, achieving higher task success rates with an improvement of 4.76%. Real-world tests demonstrate its robust zero-shot generalization feature and ability to handle diverse tasks and environments with minimal human intervention.

HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models

TL;DR

The paper tackles generalization, heterogeneity, and safety challenges in multi-robot systems by introducing HMCF, a human-in-the-loop framework that leverages LLM-based agents for per-robot reasoning and a central assistant for task allocation, augmented with human oversight to mitigate hallucinations. It integrates retrieval-augmented generation for easy onboarding of new robots and provides a user-friendly web interface for oversight, enabling robust, zero-shot generalization across diverse tasks and environments. Empirical results in BEHAVIOR-1K show a 4.76% improvement in task success rate with HMCF, and real-world deployment confirms adaptive, multi-robot coordination with minimal human intervention. While effective, the approach faces scalability and connectivity challenges for large fleets and cloud-dependent LLMs, guiding future work on scalable communication and user-centric refinements.

Abstract

Rapid advancements in artificial intelligence (AI) have enabled robots to performcomplex tasks autonomously with increasing precision. However, multi-robot systems (MRSs) face challenges in generalization, heterogeneity, and safety, especially when scaling to large-scale deployments like disaster response. Traditional approaches often lack generalization, requiring extensive engineering for new tasks and scenarios, and struggle with managing diverse robots. To overcome these limitations, we propose a Human-in-the-loop Multi-Robot Collaboration Framework (HMCF) powered by large language models (LLMs). LLMs enhance adaptability by reasoning over diverse tasks and robot capabilities, while human oversight ensures safety and reliability, intervening only when necessary. Our framework seamlessly integrates human oversight, LLM agents, and heterogeneous robots to optimize task allocation and execution. Each robot is equipped with an LLM agent capable of understanding its capabilities, converting tasks into executable instructions, and reducing hallucinations through task verification and human supervision. Simulation results show that our framework outperforms state-of-the-art task planning methods, achieving higher task success rates with an improvement of 4.76%. Real-world tests demonstrate its robust zero-shot generalization feature and ability to handle diverse tasks and environments with minimal human intervention.
Paper Structure (17 sections, 6 figures, 2 tables)

This paper contains 17 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Framework Design
  5. Human-Robots Interactive Interface
  6. LLM-Augmented Agent
  7. Robot Deployments
  8. Experiments and Evaluation
  9. Simulation and Performance Analysis
  10. Testbed
  11. Compared Models
  12. Metrics
  13. Results
  14. Ablation Study
  15. Real-World Deployment Cases
  16. ...and 2 more sections

Figures (6)

  • Figure 1: The workflow of HMCF. The system is first initialized by user input and robot configuration files, followed by task allocation and task execution. In case of failure of task execution of one or more agents, task reallocation is performed. User input can be involved in Step 3 and Step 4.
  • Figure 2: Graphical User Interface. The numbered regions represent 1) adding and querying robots, 2) robots chat list, 3) information of current chat, 4) configuration of cooperation group, 5) main chat window, and 6) user input box.
  • Figure 3: Comparison of Different Models Performance. Scenes 1 to 5 are house, store, restaurant, office, and garden, respectively.
  • Figure 4: Ablation Study: Comparison of Three HMCF Models
  • Figure 5: User-friendly configuration by providing a file of the user manual of the target robot.
  • ...and 1 more figures