Tool-RoCo: An Agent-as-Tool Self-organization Large Language Model Benchmark in Multi-robot Cooperation

TL;DR

Tool-RoCo presents a long-term, agent-as-tool benchmark for evaluating LLM-driven multi-agent cooperation across Cabinet, Pack, and Sort tasks, introducing cooperative tools and two novel metrics (CT and SO) to quantify inter-agent collaboration and self-organization. It defines four progressive paradigms—from centralized to fully self-organized—to systematically assess autonomy, tool usage, and coordination. Empirical results show larger models (e.g., GPT-5) improve tool usage and partial autonomy, but autonomously deactivating unnecessary agents remains a key bottleneck, leading to token overhead. The benchmark provides a reusable platform and a roadmap for advancing autonomous, cooperative LLM agents in multi-robot domains.

Abstract

This study proposes Tool-RoCo, a novel benchmark for evaluating large language models (LLMs) in long-term multi-agent cooperation based on RoCo, a multi-robot cooperative benchmark. Recent research on LLM-based multi-agent systems has relied on predefined orchestration, while ignoring agent autonomy. Tool-RoCo treats other agents as tools and introduces cooperative tools, leveraging tool usage to evaluate multi-agent cooperation and self-organization. Tool usage means that each agent (LLM) selects a tool from a candidate set based on the current state, receives feedback, and adjusts its selection in subsequent rounds. To evaluate different autonomy levels, we propose four LLM paradigms: (1) centralized cooperation, where a single LLM allocates tools to all agents; (2) centralized self-organization, where a central LLM autonomously activates agents while keeping others inactive; (3) decentralized cooperation, where each agent has its own LLM and calls tools based on local information; and (4) self-organization, where a randomly chosen initial agent can request collaboration, activating additional agents via tool calls. Tool-RoCo includes three multi-robot tasks, SORT, PACK, and CABINET, to measure format and parameter accuracy and agent coordination through tool usage. The results using several LLMs showed that cooperative tools accounted for only 7.09% of all tools, indicating that LLM-based agents rarely invoked others as assistants. Moreover, activation tools accounted for 96.42%, suggesting that current LLMs tend to maintain active agents while seldom deactivating them for adaptive coordination. Tool-RoCo provides a systematic benchmark to evaluate LLM autonomy and cooperation in multi-agent tasks. Code and Demo: https://github.com/ColaZhang22/Tool-Roco

Figures (3)

• Figure 1: Overview of Tool-RoCo pipeline and two kinds of tools: LLM agents perceive the current state and select appropriate tools to take action or call assistance from other agents. Then, the environment feedback and add the execution result to the next state.
• Figure 2: Three tasks and four cooperative paradigms in Tool-RoCo. The paradigms are organized along two dimensions: (i) whether tool selection is centralized by a single LLM agent or distributed across multiple agents, and (ii) whether agents are treated as tools or not. Some other agents that can be activated/deactivated for collaboration. From left(a) to right(d), the paradigms gradually increase in autonomy and complexity: centralized cooperation, centralized self-organization, decentralized cooperation, and self-organized cooperation.
• Figure 3: Comparison of prompt token consumption between centralized and decentralized paradigms. LLM agent in Centralized paradigms requires global states, actions, and candidate tool sets of all agents, whereas each agent in decentralized paradigms distributes tool selection, reducing per-agent token consumption.