What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment

TL;DR

The paper presents the Agent GPA framework, a structured evaluation paradigm for agentic AI that analyzes goals, plans, and actions using five metrics and specialized LLM judges. By applying GPA to TRAIL/GAIA and Snowflake Intelligence, the authors demonstrate comprehensive error coverage, strong alignment with human judgments, and effective localization for debugging. The framework enables actionable insights into internal agent failures—beyond final outcomes—facilitating targeted improvements in planning, tool use, and execution. This work advances scalable, interpretable, reference-free evaluation of complex, multi-step agents with practical implications for debugging and governance of autonomous systems. It also outlines reproducibility plans to foster community adoption and extension.

Abstract

We introduce the Agent GPA (Goal-Plan-Action) framework: an evaluation paradigm based on an agent's operational loop of setting goals, devising plans, and executing actions. The framework includes five evaluation metrics: Goal Fulfillment, Logical Consistency, Execution Efficiency, Plan Quality, and Plan Adherence. Logical Consistency checks that an agent's actions are consistent with its prior actions. Execution Efficiency checks whether the agent executes in the most efficient way to achieve its goal. Plan Quality checks whether an agent's plans are aligned with its goals; Plan Adherence checks if an agent's actions are aligned with its plan; and Goal Fulfillment checks that agent's final outcomes match the stated goals. Our experimental results on two benchmark datasets - the public TRAIL/GAIA dataset and an internal dataset for a production-grade data agent - show that this framework (a) provides a systematic way to cover a broad range of agent failures, including all agent errors on the TRAIL/GAIA benchmark dataset; (b) supports LLM-judges that exhibit strong agreement with human annotation, covering 80% to over 95% errors; and (c) localizes errors with 86% agreement to enable targeted improvement of agent performance.

Figures (6)

• Figure 1: GPA Evaluation LLM Judges
• Figure 2: Semantic Consistency Index (SCI) across runs for three metrics. LLM judges show higher semantic similarity in their scoring reasons for EE than PQ and LC.
• Figure 3: All GPA Judge Error Coverage Scores (0-1-2) for Dev Set
• Figure 4: All GPA Judge Error Coverage Scores (0-1-2) for Test Set
• Figure 5: Inter-rater agreement across 5 runs