LLM-Based Generalizable Hierarchical Task Planning and Execution for Heterogeneous Robot Teams with Event-Driven Replanning

TL;DR

This work addresses robust coordination of heterogeneous robot teams under dynamic conditions by proposing CoMuRoS, a hierarchical architecture that couples a centralized Task Manager LLM with decentralized Robot Brains. The Task Manager classifies tasks, allocates subtasks, and replans in response to relevant events, while each robot generates executable code from primitive skills and detects events onboard. Hardware and simulation experiments demonstrate event-driven replanning, autonomous recovery, and emergent human–robot collaboration, supported by a textual benchmark evaluating generalization across tasks, scenarios, and LLM backends. The system achieves high performance in planning correctness, executability, and coordination, and uniquely demonstrates runtime, event-driven replanning on physical robots with scalable ROS2 integration. These results indicate strong potential for flexible, resilient multi-robot and human–multirobot collaboration in real-world settings.

Abstract

This paper introduces CoMuRoS (Collaborative Multi-Robot System), a generalizable hierarchical architecture for heterogeneous robot teams that unifies centralized deliberation with decentralized execution, and supports event-driven replanning. A Task Manager LLM interprets natural-language goals, classifies tasks, and allocates subtasks using static rules plus dynamic contexts (task, history, robot and task status, and events).Each robot runs a local LLM that composes executable Python code from primitive skills (ROS2 nodes, policies), while onboard perception (VLMs/image processing) continuously monitors events and classifies them into relevant or irrelevant to the task. Task failures or user intent changes trigger replanning, allowing robots to assist teammates, resume tasks, or request human help. Hardware studies demonstrate autonomous recovery from disruptive events, filtering of irrelevant distractions, and tightly coordinated transport with emergent human-robot cooperation (e.g., multirobot collaborative object recovery success rate: 9/10, coordinated transport: 8/8, human-assisted recovery: 5/5).Simulation studies show intention-aware replanning. A curated textual benchmark spanning 22 scenarios (3 tasks each, around 20 robots) evaluates task allocation, classification, IoU, executability, and correctness, with high average scores (e.g., correctness up to 0.91) across multiple LLMs, a separate replanning set (5 scenarios) achieves 1.0 correctness. Compared with prior LLM-based systems, CoMuRoS uniquely demonstrates runtime, event-driven replanning on physical robots, delivering robust, flexible multi-robot and human-robot collaboration.

Figures (13)

• Figure 1: Architectural Diagram of CoMuRoS
• Figure 2: Task Manager Prompt
• Figure 3: Chat Interface for human robot collaboration.
• Figure 4: Demonstration of event-driven replanning, incomplete task resumption, and emergence of cooperation where robots assist one another.
• Figure 5: Demonstration of ignoring irrelevant events.