Understanding Multi-Agent LLM Frameworks: A Unified Benchmark and Experimental Analysis

TL;DR

This work tackles the problem of understanding how architecture, not just LLM quality, shapes the performance of multi-agent LLM systems. It introduces an architectural taxonomy and MAFBench, a unified evaluation suite that standardizes the pipeline across memory, planning, specialization, tool use, and coordination while fixing the underlying models. The empirical study demonstrates that framework-level design choices can cause latency to vary by over $100\times$, planning accuracy to drop up to 30%, and coordination success to fall from over 90% to below 30%, highlighting architecture as the primary performance lever. The paper contributes a formal taxonomy, the MAFBench suite, first controlled comparisons across design choices, and actionable design principles, paving the way for principled, scalable, and automated design of multi-agent agentic systems.

Abstract

Multi-agent LLM frameworks are widely used to accelerate the development of agent systems powered by large language models (LLMs). These frameworks impose distinct architectural structures that govern how agents interact, store information, and coordinate tasks. However, their impact on system performance remains poorly understood. This gap is critical, as architectural choices alone can induce order-of-magnitude differences in latency and throughput, as well as substantial variation in accuracy and scalability. Addressing this challenge requires (i) jointly evaluating multiple capabilities, such as orchestration overhead, memory behavior, planning, specialization, and coordination, and (ii) conducting these evaluations under controlled, framework-level conditions to isolate architectural effects. Existing benchmarks focus on individual capabilities and lack standardized framework-level evaluation. We address these limitations by (i) introducing an architectural taxonomy for systematically comparing multi-agent LLM frameworks along fundamental dimensions, and (ii) developing MAFBench, a unified evaluation suite that integrates existing benchmarks under a standardized execution pipeline. Using MAFBench, we conduct a controlled empirical study across several widely used frameworks. Our results show that framework-level design choices alone can increase latency by over 100x, reduce planning accuracy by up to 30%, and lower coordination success from above 90% to below 30%. Finally, we translate our findings into concrete architectural design principles and framework selection guidance, and outline promising future research directions.

Figures (9)

• Figure 1: Characteristics of StableToolBench, showing the distributions of (a) query length, (b) number of APIs per tool, and (c) query counts across tool categories.
• Figure 2: Framework overhead on a trivial task (“What is 2+2?”) over 50 trials. Frameworks are ordered by p50 latency.
• Figure 3: Runtime as a function of context window for LangGraph (left) and OpenAI SDK (right) across the four task.
• Figure 4: Consensus experiment outcomes across different topologies and network sizes ($n=4$ to $100$). Abbreviations denote topology classes: SW = Small-World, SF = Scale-Free, DT = Delaunay (geometric), Seq = Sequential role-based pipeline, Hier = Hierarchical role-based orchestration, and All = fully connected (all-to-all) communication. Each subfigure visualizes the final agent states and their communication links. Green links indicate agreement between two agents, while red links indicate disagreement or conflict.
• Figure 5: Coloring experiment outcomes across different network sizes ($n=4$ to $100$). Each subfigure shows the final group assignments of agents. Valid assignments (neighbors in different groups) are shown in green, while conflicts (neighbors in the same group) are shown in red.