SayCoNav: Utilizing Large Language Models for Adaptive Collaboration in Decentralized Multi-Robot Navigation

TL;DR

SayCoNav tackles decentralized coordination for heterogeneous robot teams in large unknown environments by using LLMs to automatically generate collaboration strategies grounded in each robot's capabilities. It introduces a three-level decentralized planning architecture (Top-Level Global Planner, Middle-Level Local Planner, Graph Generator/Bottom-Level Action Planner) enabling dynamic, decentralized decision-making and adaptation to changing conditions. Experimental results on MultiON within ProcTHOR demonstrate up to a 44.28% improvement in search efficiency over a SayNav baseline and show effective adaptation to team composition and missions. The work advances practical multi-robot navigation by enabling heterogeneous teams to collaboratively explore and locate multiple objects in unknown spaces, though it notes limitations in complex manipulation coordination and LLM hallucinations.

Abstract

Adaptive collaboration is critical to a team of autonomous robots to perform complicated navigation tasks in large-scale unknown environments. An effective collaboration strategy should be determined and adapted according to each robot's skills and current status to successfully achieve the shared goal. We present SayCoNav, a new approach that leverages large language models (LLMs) for automatically generating this collaboration strategy among a team of robots. Building on the collaboration strategy, each robot uses the LLM to generate its plans and actions in a decentralized way. By sharing information to each other during navigation, each robot also continuously updates its step-by-step plans accordingly. We evaluate SayCoNav on Multi-Object Navigation (MultiON) tasks, that require the team of the robots to utilize their complementary strengths to efficiently search multiple different objects in unknown environments. By validating SayCoNav with varied team compositions and conditions against baseline methods, our experimental results show that SayCoNav can improve search efficiency by at most 44.28% through effective collaboration among heterogeneous robots. It can also dynamically adapt to the changing conditions during task execution.

Figures (4)

• Figure 1: A SayCoNav example: Three heterogeneous robots use LLMs to define and to execute the collaboration strategy based on their skills, for searching two target objects in a new house.
• Figure 2: The SayCoNav framework for a single robot.
• Figure 3: Prompts for top-level global planner (left) and middle-level local planner (right). The global planner prompt includes team task statement (blue), robot skills and conditions (red), collaboration strategy request (purple), and tips (green). The local planner prompt has four components prior to the request to LLM: feedback, summary, local scene graph, and communication messages.
• Figure 4: Example of failure cases where a small pencil is present inside a box (left) and a small egg is inside a fridge (right).