Toward Architecture-Aware Evaluation Metrics for LLM Agents

TL;DR

The paper tackles the problem of fragmented, model-centric evaluation of LLM-based agents by proposing an architecture-aware framework that links observable behaviors to specific architectural components (e.g., planners, memory, tool routers) and to appropriate metrics. It introduces a design-science mapping approach that ties behaviors and components to existing evaluation metrics from tools like LangSmith, DeepEval, and Opik, without proposing new metrics. The core contributions are a formal agent component taxonomy with behavior signals and an architecture-informed evaluation method, validated on real-world agents such as SWE-Agent and MetaGPT to demonstrate improved diagnostic power and reproducibility. This framework promises more targeted, transparent, and actionable evaluation, reducing metric arbitrariness and enhancing reliability in software engineering tasks involving LLM-based agents.

Abstract

LLM-based agents are becoming central to software engineering tasks, yet evaluating them remains fragmented and largely model-centric. Existing studies overlook how architectural components, such as planners, memory, and tool routers, shape agent behavior, limiting diagnostic power. We propose a lightweight, architecture-informed approach that links agent components to their observable behaviors and to the metrics capable of evaluating them. Our method clarifies what to measure and why, and we illustrate its application through real world agents, enabling more targeted, transparent, and actionable evaluation of LLM-based agents.