WebGraphEval: Multi-Turn Trajectory Evaluation for Web Agents using Graph Representation

TL;DR

WebGraphEval addresses the limitation of outcome-only evaluation by proposing a graph-based framework that aggregates diverse web-agent trajectories into a single consensus graph, enabling multi-path, cross-model, and efficiency-aware analysis. The methodology combines trajectory canonicalization, two-pass consensus graph construction, and dual reward mechanisms (reward backpropagation with a discount factor $\gamma$ and edge-based success weighting) to reveal structual decision points and shared strategies. Empirical results on 4,768 trajectories across 812 tasks from six agents demonstrate meaningful cross-model variation, efficiency patterns, and complementary strengths, with a reliable annotation pipeline and a visualization tool. The framework provides a practical, scalable approach for advancing web agent evaluation and can inform online decision-making and transfer learning across tasks and interfaces.

Abstract

Current evaluation of web agents largely reduces to binary success metrics or conformity to a single reference trajectory, ignoring the structural diversity present in benchmark datasets. We present WebGraphEval, a framework that abstracts trajectories from multiple agents into a unified, weighted action graph. This representation is directly compatible with benchmarks such as WebArena, leveraging leaderboard runs and newly collected trajectories without modifying environments. The framework canonically encodes actions, merges recurring behaviors, and applies structural analyses including reward propagation and success-weighted edge statistics. Evaluations across thousands of trajectories from six web agents show that the graph abstraction captures cross-model regularities, highlights redundancy and inefficiency, and identifies critical decision points overlooked by outcome-based metrics. By framing web interaction as graph-structured data, WebGraphEval establishes a general methodology for multi-path, cross-agent, and efficiency-aware evaluation of web agents.

Figures (14)

• Figure 1: Overview of the WebGraphEval pipeline. Trajectories from multiple agents are first pre-processed and canonicalized into standardized action units. These are merged into a directed action graph, where nodes represent actions and edges represent observed transitions. The graph is then analyzed with reward backpropagation and success-weighted edge classification, and the resulting structure supports multi-dimensional evaluation of agent efficiency, redundancy, and strategy overlap.
• Figure 2: Action necessity across frameworks and learning dynamics. (Left) Necessity rates of six frameworks, showing variation from 72.9% to 82.0%. (Right) Learning curve showing how necessity rates improve with repeated task attempts, suggesting necessity is a learnable signal for agent optimization.
• Figure 3: Trajectory efficiency and success patterns. (Left) Success rates follow an inverted-U curve with respect to trajectory length, peaking at medium trajectories (6–10 steps, 53.4%). (Right) Step inflation by task complexity shows a counter-intuitive trend: agents inflate simple tasks the most (3.18$\times$), while complex tasks approach or even beat near-optimal paths due to survival bias.
• Figure 4: Core necessity metrics across four dimensions: overall, action type, complexity, and confidence.
• Figure 5: Supplementary behavioral patterns: (a) necessity varies across trajectory phases, with peak efficiency in the middle; (b) action type distribution reveals clicks dominate overall interactions.