Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI

Manik Rana, Calissa Man, Anotida Expected Msiiwa, Jeffrey Paine, Kevin Zhu, Sunishchal Dev, Vasu Sharma, Ahan M R

TL;DR

AgentChangeBench tackles the problem of evaluating conversational agents under mid-dialog goal shifts by introducing a multi-domain benchmark (banking, airline, retail) with five user personas and explicit goal-shift sequences. It departs from traditional pass@k by coupling four metrics—Task Success Rate (TSR), Tool Usage Efficiency (TUE), Tool-Call Redundancy Ratio (TCRR), and Goal-Shift Recovery Time (GSRT)—to measure effectiveness, efficiency, redundancy, and adaptation latency. An empirical study across GPT-4o, Claude-3.7-Sonnet, and Gemini-2.5-Flash reveals substantial divergences in robustness, showing that high raw accuracy does not guarantee quick or efficient adaptation to changing goals. The benchmark enables reproducible evaluation and provides guidance for deploying robust enterprise agents, with clear avenues for future expansion to more domains, tools, and multilingual settings.

Abstract

Goal changes are a defining feature of real world multi-turn interactions, yet current agent benchmarks primarily evaluate static objectives or one-shot tool use. We introduce AgentChangeBench, a benchmark explicitly designed to measure how tool augmented language model agents adapt to mid dialogue goal shifts across three enterprise domains. Our framework formalizes evaluation through four complementary metrics: Task Success Rate (TSR) for effectiveness, Tool Use Efficiency (TUE) for reliability, Tool Call Redundancy Rate (TCRR) for wasted effort, and Goal-Shift Recovery Time (GSRT) for adaptation latency. AgentChangeBench comprises 2,835 task sequences and five user personas, each designed to trigger realistic shift points in ongoing workflows. Using this setup, we evaluate several frontier models and uncover sharp contrasts obscured by traditional $\text{pass}@k$ scores: for example, GPT-4o reaches $92.2\%$ recovery on airline booking shifts while Gemini collapses to $48.6\%$, and retail tasks show near perfect parameter validity yet redundancy rates above $80\%$, revealing major inefficiencies. These findings demonstrate that high raw accuracy does not imply robustness under dynamic goals, and that explicit measurement of recovery time and redundancy is essential. AgentChangeBench establishes a reproducible testbed for diagnosing and improving agent resilience in realistic enterprise settings.

AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI

TL;DR

AgentChangeBench tackles the problem of evaluating conversational agents under mid-dialog goal shifts by introducing a multi-domain benchmark (banking, airline, retail) with five user personas and explicit goal-shift sequences. It departs from traditional pass@k by coupling four metrics—Task Success Rate (TSR), Tool Usage Efficiency (TUE), Tool-Call Redundancy Ratio (TCRR), and Goal-Shift Recovery Time (GSRT)—to measure effectiveness, efficiency, redundancy, and adaptation latency. An empirical study across GPT-4o, Claude-3.7-Sonnet, and Gemini-2.5-Flash reveals substantial divergences in robustness, showing that high raw accuracy does not guarantee quick or efficient adaptation to changing goals. The benchmark enables reproducible evaluation and provides guidance for deploying robust enterprise agents, with clear avenues for future expansion to more domains, tools, and multilingual settings.

Abstract

Goal changes are a defining feature of real world multi-turn interactions, yet current agent benchmarks primarily evaluate static objectives or one-shot tool use. We introduce AgentChangeBench, a benchmark explicitly designed to measure how tool augmented language model agents adapt to mid dialogue goal shifts across three enterprise domains. Our framework formalizes evaluation through four complementary metrics: Task Success Rate (TSR) for effectiveness, Tool Use Efficiency (TUE) for reliability, Tool Call Redundancy Rate (TCRR) for wasted effort, and Goal-Shift Recovery Time (GSRT) for adaptation latency. AgentChangeBench comprises 2,835 task sequences and five user personas, each designed to trigger realistic shift points in ongoing workflows. Using this setup, we evaluate several frontier models and uncover sharp contrasts obscured by traditional scores: for example, GPT-4o reaches recovery on airline booking shifts while Gemini collapses to , and retail tasks show near perfect parameter validity yet redundancy rates above , revealing major inefficiencies. These findings demonstrate that high raw accuracy does not imply robustness under dynamic goals, and that explicit measurement of recovery time and redundancy is essential. AgentChangeBench establishes a reproducible testbed for diagnosing and improving agent resilience in realistic enterprise settings.
Paper Structure (49 sections, 5 equations, 3 figures, 11 tables)

This paper contains 49 sections, 5 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Release.
  3. Related Work
  4. Methodology
  5. Dataset Design
  6. Task Generation
  7. User Personas
  8. Task Schema
  9. Examples.
  10. User/agent control.
  11. Evaluation Harness
  12. Results and Evaluation Metrics
  13. Motivation and Problem Statement
  14. Key Contributions
  15. Evaluation Metrics
  16. ...and 34 more sections

Figures (3)

  • Figure 1: GSRT by model and domain. Turns after a user goal shift to acknowledgment, first relevant tool call, and completion, lower is better. Aggregated over all tasks and personas in banking, retail, and airline for GPT-4o, Claude-3.7-Sonnet, and Gemini-2.5-Flash; labels show recovery rate, percent of shifts acknowledged without human transfer. Source: Appendix Tables \ref{['tab:efficiency_full']}, \ref{['tab:gs_new_only']}.
  • Figure 2: TUE components.Top: Tool Correctness (TC) by simulation run. Boxes are light coral; the dotted orange line marks a tail threshold at $0.7$. Middle: Parameter Validity (PV) by simulation run. The dotted red line marks the ceiling at $0.95$. Bottom: TC aggregated across tasks (each point is a task mean over runs). Box is sky blue; the dotted orange line again marks $0.7$. Axes are clipped to $[0,1.02]$.
  • Figure 3: System architecture highlighting goal-shift capabilities (orange) and enhanced metrics (red) built upon $\tau^2$-bench. Goal-shift capabilities are integrated into the core framework rather than being standalone components.