Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability

Tie Ma, Yixi Chen, Vaastav Anand, Alessandro Cornacchia, Amândio R. Faustino, Guanheng Liu, Shan Zhang, Hongbin Luo, Suhaib A. Fahmy, Zafar A. Qazi, Marco Canini

TL;DR

MAESTRO introduces an open-source evaluation suite for LLM-based MAS that standardizes configuration, supports native and adapter-based integration, and exports comprehensive system telemetry. It demonstrates, across 12 MAS and two evaluation suites, that MAS executions are structurally stable yet temporally variable, producing high run-to-run variance in latency, cost, and reliability. The study finds that MAS architecture predominantly shapes resource profiles, reproducibility, and the cost latency accuracy trade-off, often outweighing backend model or tool choices. These results provide empirical guidance for designing, optimizing, and benchmarking reliable agentic systems and highlight the need for standardized observability contracts in MAS benchmarks.

Abstract

We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS executions can be structurally stable yet temporally variable, leading to substantial run-to-run variance in performance and reliability. We further find that MAS architecture is the dominant driver of resource profiles, reproducibility, and cost-latency-accuracy trade-off, often outweighing changes in backend models or tool settings. Overall, MAESTRO enables systematic evaluation and provides empirical guidance for designing and optimizing agentic systems.

MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability

TL;DR

MAESTRO introduces an open-source evaluation suite for LLM-based MAS that standardizes configuration, supports native and adapter-based integration, and exports comprehensive system telemetry. It demonstrates, across 12 MAS and two evaluation suites, that MAS executions are structurally stable yet temporally variable, producing high run-to-run variance in latency, cost, and reliability. The study finds that MAS architecture predominantly shapes resource profiles, reproducibility, and the cost latency accuracy trade-off, often outweighing backend model or tool choices. These results provide empirical guidance for designing, optimizing, and benchmarking reliable agentic systems and highlight the need for standardized observability contracts in MAS benchmarks.

Abstract

We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS executions can be structurally stable yet temporally variable, leading to substantial run-to-run variance in performance and reliability. We further find that MAS architecture is the dominant driver of resource profiles, reproducibility, and cost-latency-accuracy trade-off, often outweighing changes in backend models or tool settings. Overall, MAESTRO enables systematic evaluation and provides empirical guidance for designing and optimizing agentic systems.
Paper Structure (27 sections, 12 figures, 3 tables)

This paper contains 27 sections, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Anatomy of an LLM-based MAS
  4. Limitations of existing benchmarks
  5. MAESTRO
  6. Benchmark design
  7. MAESTRO architecture
  8. MAS instances preparation.
  9. MAS instances taxonomy
  10. MAS example suites studied
  11. Case studies
  12. Methodology
  13. What are the systems usage patterns and implications?
  14. How stable are MAS call graphs, and what factors influence their variability?
  15. How do different agent architectures affect task performance and stability?
  16. ...and 12 more sections

Figures (12)

  • Figure 1: MAESTRO architecture overview.
  • Figure 2: Two ways to prepare MAS instances for MAESTRO, note that MAESTRO ships with a set of built-in MAS instances that can be used and compared directly.
  • Figure 3: Solving the same given tasks with 3 different MAS architectures.
  • Figure 4: CPU and memory usage across different examples.
  • Figure 5: Communication usage across different architectures.
  • ...and 7 more figures