AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents

TL;DR

AgentArcEval tackles the challenge of evaluating architecture for foundation-model based agents, arguing that traditional design-time methods like ATAM fail to address goal-driven autonomy, runtime evolution, and guardrail-driven tradeoffs. The authors propose an ATAM-inspired method tailored to agents, including a catalogue of agent-specific general scenarios and a Luna tax copilot case study to demonstrate practical applicability. The approach combines hands-on project experience with a systematic literature review to produce a structured, scenario-based evaluation workflow, spanning design-time and runtime phases, with guardrails and continuous risk monitoring. The Luna case study shows concrete architectural adjustments—such as AgentOps logging, memory-based adaptability, and transparent sourcing—that improve accuracy, adaptability, efficiency, observability, and privacy, illustrating the method’s value for real-world FM-based agent systems. They also position AgentArcEval as a living, community-driven framework intended to evolve with practice and broad adoption across agentic systems.

Abstract

The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna.

Figures (14)

• Figure 1: Agent Reference Architecture lu2024towards.
• Figure 2: The process of AgentArcEval.
• Figure 3: Accuracy general scenario.
• Figure 4: Adaptability general scenario.
• Figure 5: Efficiency general scenario.