Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MASEval: Extending Multi-Agent Evaluation from Models to Systems

Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun, Seong Joon Oh, Martin Gubri

TL;DR

Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, it is found that framework choice matters as much as model choice.

Abstract

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

MASEval: Extending Multi-Agent Evaluation from Models to Systems

TL;DR

Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, it is found that framework choice matters as much as model choice.

Abstract

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.
Paper Structure (16 sections, 3 figures, 5 tables)

This paper contains 16 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Architecture
  4. Design Principles
  5. Module Architecture
  6. Core Abstractions
  7. Benchmark Lifecycle
  8. Key Capabilities
  9. Key Features
  10. System-Level Benchmarking
  11. Experimental Setup
  12. Results
  13. Implementation Effort
  14. Conclusion
  15. Benchmarks and Frameworks
  16. ...and 1 more sections

Figures (3)

  • Figure 1: MASEval provides a unified evaluation layer that enables framework-agnostic, system-level comparison across any agent framework and benchmark.
  • Figure 2: MASEval adopts a "Bring Your Own" (BYO) philosophy. Users implement custom components by extending MASEval's abstract base classes; the core runtime orchestrates execution and collects traces. This enables maximum flexibility while minimizing boilerplate code.
  • Figure 3: Benchmark task lifecycle with flexible execution. The outer loop iterates over tasks; the inner loop handles repetitions. The Execute phase shows agent-user interaction as a flexible bidirectional loop.