Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which LLM Multi-Agent Protocol to Choose?

Hongyi Du, Jiaqi Su, Jisen Li, Lijie Ding, Yingxuan Yang, Peixuan Han, Xiangru Tang, Kunlun Zhu, Jiaxuan You

TL;DR

This work introduces ProtocolBench, the first comprehensive, protocol-agnostic benchmark for evaluating LLM-based multi-agent communication protocols across four scenarios, revealing that protocol choice meaningfully impacts task success, latency, overhead, and resilience. It then proposes ProtocolRouter, a learnable per-module protocol selector that operates with adapters to translate between protocols without altering business semantics, and demonstrates its ability to improve recovery times and task outcomes in targeted scenarios. Through ProtocolBench and ProtocolRouterBench, the authors provide standardized evaluation artifacts, datasets, and railway-tested metrics to enable reliable, scalable protocol analysis. The results underscore that no single protocol dominates universally; a dynamic, scenario-aware routing approach can yield tangible gains in reliability and efficiency, guiding practitioners toward principled protocol design and selection. The work has practical significance for production MAS, offering reproducible benchmarks and tooling to optimize coordination, security, and performance at scale.

Abstract

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.

Which LLM Multi-Agent Protocol to Choose?

TL;DR

This work introduces ProtocolBench, the first comprehensive, protocol-agnostic benchmark for evaluating LLM-based multi-agent communication protocols across four scenarios, revealing that protocol choice meaningfully impacts task success, latency, overhead, and resilience. It then proposes ProtocolRouter, a learnable per-module protocol selector that operates with adapters to translate between protocols without altering business semantics, and demonstrates its ability to improve recovery times and task outcomes in targeted scenarios. Through ProtocolBench and ProtocolRouterBench, the authors provide standardized evaluation artifacts, datasets, and railway-tested metrics to enable reliable, scalable protocol analysis. The results underscore that no single protocol dominates universally; a dynamic, scenario-aware routing approach can yield tangible gains in reliability and efficiency, guiding practitioners toward principled protocol design and selection. The work has practical significance for production MAS, offering reproducible benchmarks and tooling to optimize coordination, security, and performance at scale.

Abstract

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.
Paper Structure (127 sections, 4 equations, 4 figures, 8 tables)

This paper contains 127 sections, 4 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks and Multi-agent frameworks.
  4. Agent protocols and communication mechanisms.
  5. ProtocolBench: A Systematic Evaluation of Agent Protocols
  6. ProtocolBench scenarios
  7. System design and evaluation
  8. ProtocolRouter: A Task-dependent Selection of Protocols
  9. ProtocolRouter Design
  10. Goals.
  11. Inputs.
  12. Outputs and runtime.
  13. Non-goals and limits.
  14. ProtocolRouterBench: Extending ProtocolBench to Evaluate Multi-agent Protocol Routers
  15. Objective and evaluation modes.
  16. ...and 112 more sections

Figures (4)

  • Figure 1: Overview of ProtocolBench and ProtocolRouter. To understand the tradeoff across existing LLM multi-agent protocols, we first design ProtocolBench that covers four core evaluation dimensions, then propose ProtocolRouter to help users select the optimal protocol.
  • Figure 2: Illustration of four multi-agent scenarios evaluated in this work.
  • Figure 3: ProtocolRouter overview. A scenario-aware selector (top) (Appendix C.10) outputs a structured plan with one protocol per module. The agent protocol adapters and connects agents (bottom); cross-protocol links use stateless encode/decode bridges without shared session state.
  • Figure 4: Case study: ProtocolRouter assigns protocols per module for the GAIA metro-counting task, enabling each module to run on its most suitable protocol (e.g., Agora for upstream discovery/compute and ACP for the final commit). This per-module assignment yields an overall accuracy that exceeds the single-protocol A2A baseline by 6.5%.