Benchmarking LLMs' Swarm intelligence

TL;DR

This work introduces SwarmBench, a benchmark to evaluate LLM-driven agents operating under strict swarm-like decentralization in a configurable 2D grid. It defines five canonical multi-agent coordination tasks and a discrete physics engine to model cooperative pushing, with agents restricted to local perception and local communication. Zero-shot evaluations across multiple contemporary LLMs reveal significant task-dependent performance and limited long-range planning, with emergent coordination driven more by physical group dynamics than by explicit communication. The authors provide an open-source toolkit including environments, prompts, datasets, and evaluation scripts to study emergent collective behavior under information decentralization and outline a path toward more robust, swarm-capable LLMs.

Abstract

Large Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) when operating under strict swarm-like constraints-limited local perception and communication-remains largely unexplored. Existing benchmarks often do not fully capture the unique challenges of decentralized coordination when agents operate with incomplete spatio-temporal information. To bridge this gap, we introduce SwarmBench, a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of LLMs acting as decentralized agents. SwarmBench features five foundational MAS coordination tasks (Pursuit, Synchronization, Foraging, Flocking, Transport) within a configurable 2D grid environment, forcing agents to rely solely on local sensory input ($k\times k$ view) and local communication. We propose metrics for coordination effectiveness and analyze emergent group dynamics. Zero-shot evaluations of leading LLMs (e.g., deepseek-v3, o4-mini) reveal significant task-dependent performance variations. While some rudimentary coordination is observed, our results indicate that current LLMs significantly struggle with robust long-range planning and adaptive strategy formation under the uncertainty inherent in these decentralized scenarios. Assessing LLMs under such swarm-like constraints is crucial for understanding their utility in future decentralized intelligent systems. We release SwarmBench as an open, extensible toolkit-built on a customizable physical system-providing environments, prompts, evaluation scripts, and comprehensive datasets. This aims to foster reproducible research into LLM-based MAS coordination and the theoretical underpinnings of emergent collective behavior under severe informational decentralization. Our code repository is available at https://github.com/x66ccff/swarmbench.

Figures (107)

• Figure 1: Swarm Intelligence: Natural Inspiration and SwarmBench Tasks.Top row: Examples of collective behavior in nature driven by local interactions: a. Cooperative wolf pursuit, b. firefly synchronization, c. ant foraging Reid2015Powell2021, d. bird flocking reynolds1987flocks_Flocks_herds_and_schools, and e. cooperative ant transport. Bottom row: Corresponding abstract tasks simulated in SwarmBench's 2D grid environment, depicting agents (represented by colored squares) facing analogous coordination challenges involving the prey (P), food (F), nests (N), and obstacles (B), constrained by walls (W). Agents rely solely on local perception and communication, providing a testbed for emergent decentralized coordination. Detailed SwarmBench environment definition and examples can be found in Appendix \ref{['app:swarmbench_details']} and Appendix \ref{['app:examples']} respectively. Replay videos can be found in Supplementary Materials (see Supplementary Videos)
• Figure 2: Conceptual Architecture of SwarmBench. The diagram shows SwarmBench's modular design. It orchestrates task, environment, physics, LLM-agents, and logger to benchmark LLM swarm intelligence, generate agent-environment interaction datasets, and serve as a swarm intelligence RLVR (Reinforcement Learning with Verifiable Rewards) environment. Our codes can be found in Supplementary Materials (see Supplementary Codes 1)
• Figure 3: Metrics for gemini-2.0-flash on the Pursuit task. This figure illustrates score progression and various group dynamic metrics. Detailed definitions for all metrics are provided in Appendix \ref{['app:group_dynamics_metrics']}. Comprehensive visualizations for all evaluated models and tasks can be found in Fig. \ref{['fig:mc_claude-3.5-haiku_synchronize']}--\ref{['fig:mc_qwq-32b_transport']}, Appendix \ref{['app:model_specific_dynamics_viz']}. The shaded area represents the standard deviation of five simulation runs on this task.
• Figure 4: Overview of LLM Performance on SwarmBench Tasks. Average scores by LLMs across five core tasks. Bars: mean score over 5 runs. The difficulty of these five tasks varies. It is worth noting that even for the seemingly simple Transport task (i.e., Moving a large, irregularly shaped obstacle blocking the map exit by following appropriate steps, see Fig. \ref{['fig:transport']}), only o4-mini and deepseek-r1 were able to achieve a non-zero average score. Performance varies significantly by model and challenge. Details in Table \ref{['tab:detailed_scores']}, Appendix \ref{['app:detailed_scores']}.
• Figure 5: LLM Score Progression on SwarmBench Tasks Over Time. Average task score accumulation over 100 rounds. Lines: mean score trajectory; shaded areas: std. dev. Illustrates performance dynamics.