Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compositional Coordination for Multi-Robot Teams with Large Language Models

Zhehui Huang, Guangyao Shi, Yuwei Wu, Vijay Kumar, Gaurav S. Sukhatme

TL;DR

LAN2CB introduces a language-to-coordinated-behavior framework that converts natural-language mission descriptions into executable Python code for robot teams. The system splits into Mission Analysis (NL-to-behavior-tree decomposition) and Code Generation (synthesis of robot control code using a structured knowledge base and a meta-behavior library), enabling automatic execution and replanning through an iterative execution loop. A dedicated NL mission dataset and comprehensive experiments—spanning nine missions, simulations, and real-world demos—demonstrate robust, flexible coordination across diverse tasks and highlight the importance of standardized prompting. The work promises to reduce manual engineering effort and enhance generalization for large, heterogeneous robot teams in dynamic environments.

Abstract

Multi-robot coordination has traditionally relied on a mission-specific and expert-driven pipeline, where natural language mission descriptions are manually translated by domain experts into mathematical formulation, algorithm design, and executable code. This conventional process is labor-intensive, inaccessible to non-experts, and inflexible to changes in mission requirements. Here, we propose LAN2CB (Language to Collective Behavior), a novel framework that leverages large language models (LLMs) to streamline and generalize the multi-robot coordination pipeline. LAN2CB transforms natural language (NL) mission descriptions into executable Python code for multi-robot systems through two core modules: (1) Mission Analysis, which parses mission descriptions into behavior trees, and (2) Code Generation, which leverages the behavior tree and a structured knowledge base to generate robot control code. We further introduce a dataset of natural language mission descriptions to support development and benchmarking. Experiments in both simulation and real-world environments demonstrate that LAN2CB enables robust and flexible multi-robot coordination from natural language, significantly reducing manual engineering effort and supporting broad generalization across diverse mission types. Website: https://sites.google.com/view/lan-cb

Compositional Coordination for Multi-Robot Teams with Large Language Models

TL;DR

LAN2CB introduces a language-to-coordinated-behavior framework that converts natural-language mission descriptions into executable Python code for robot teams. The system splits into Mission Analysis (NL-to-behavior-tree decomposition) and Code Generation (synthesis of robot control code using a structured knowledge base and a meta-behavior library), enabling automatic execution and replanning through an iterative execution loop. A dedicated NL mission dataset and comprehensive experiments—spanning nine missions, simulations, and real-world demos—demonstrate robust, flexible coordination across diverse tasks and highlight the importance of standardized prompting. The work promises to reduce manual engineering effort and enhance generalization for large, heterogeneous robot teams in dynamic environments.

Abstract

Multi-robot coordination has traditionally relied on a mission-specific and expert-driven pipeline, where natural language mission descriptions are manually translated by domain experts into mathematical formulation, algorithm design, and executable code. This conventional process is labor-intensive, inaccessible to non-experts, and inflexible to changes in mission requirements. Here, we propose LAN2CB (Language to Collective Behavior), a novel framework that leverages large language models (LLMs) to streamline and generalize the multi-robot coordination pipeline. LAN2CB transforms natural language (NL) mission descriptions into executable Python code for multi-robot systems through two core modules: (1) Mission Analysis, which parses mission descriptions into behavior trees, and (2) Code Generation, which leverages the behavior tree and a structured knowledge base to generate robot control code. We further introduce a dataset of natural language mission descriptions to support development and benchmarking. Experiments in both simulation and real-world environments demonstrate that LAN2CB enables robust and flexible multi-robot coordination from natural language, significantly reducing manual engineering effort and supporting broad generalization across diverse mission types. Website: https://sites.google.com/view/lan-cb

Paper Structure

This paper contains 17 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Robot Coordination
  4. Integrating LLMs in Robotics
  5. Single-Robot Integration
  6. Team-Level Integration
  7. Problem Statement
  8. Method
  9. Mission Analysis
  10. Code Generation
  11. Execution
  12. Experiments
  13. Performance Across Diverse Missions
  14. Impact of standardized template
  15. Realistic Simulated Scenarios
  16. ...and 2 more sections

Figures (5)

  • Figure 1: LAN2CB is an LLM-assisted multi-robot framework that converts mission descriptions into structured representations (e.g., behavior trees), assigns roles and priorities to each robot, and automatically generates executable code to accomplish complex missions. A mission change does not require human intervention since new code is generated automatically. NL means natural language. user- or expert-provided content. LLM-generated mission representations. LLM-generated code.
  • Figure 2: A comprehensive demonstration of LAN2CB. human-provided content. LLM-generated mission descriptions. LLM-generated code.
  • Figure 3: Quantitative results for nine multi-robot missions. Please check the https://sites.google.com/view/lan-cb/home for more details.
  • Figure 4: Realistic-setting experiments. idle, running, finished.
  • Figure 5: Real-world experiments. idle, running, finished.