AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents

TL;DR

AURA introduces a domain-agnostic, POMDP-based framework for evaluating user satisfaction in interactive planning agents, addressing the shortcomings of task-completion metrics. It decomposes the agent's behavior into atomic criteria—State Consistency, Tool Efficiency, Observation Alignment, Policy Alignment, and Task Completion—plus interaction-pattern metrics to diagnose multi-turn decision pipelines. Experiments across TravelPlanner and taubench benchmarks show that intermediate behaviors strongly influence user satisfaction, and human studies corroborate that stage-aware metrics align with preferences beyond final outcomes. The work also explores mixed-agent deployment and the reliability of user simulators, highlighting directions for more robust, interpretable evaluation of planning agents.

Abstract

The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task planning agents have become especially prominent in realistic scenarios involving complex internal pipelines, such as context understanding, tool management, and response generation. However, existing benchmarks predominantly evaluate agent performance based on task completion as a proxy for overall effectiveness. We hypothesize that merely improving task completion is misaligned with maximizing user satisfaction, as users interact with the entire agentic process and not only the end result. To address this gap, we propose AURA, an Agent-User inteRaction Assessment framework that conceptualizes the behavioral stages of interactive task planning agents. AURA offers a comprehensive assessment of agent through a set of atomic LLM evaluation criteria, allowing researchers and practitioners to diagnose specific strengths and weaknesses within the agent's decision-making pipeline. Our analyses show that agents excel in different behavioral stages, with user satisfaction shaped by both outcomes and intermediate behaviors. We also highlight future directions, including systems that leverage multiple agents and the limitations of user simulators in task planning.

Figures (5)

• Figure 1: AURA provides unified, atomic, and domain-agnostic criteria for assessing user satisfaction of interactive planning agents, extending beyond conventional evaluation protocols that focus solely on task completion.
• Figure 2: Decision theory-based taxonomy of evaluation metrics in AURA. As discussed in walker-etal-1997-paradise, it should be cautious to generalize the original taxonomy to different agents and tasks. Following this, we provide five distinct interpretations of each element, which will be described in \ref{['sec:protocoltask']}.
• Figure 3: Human study results examining preferred interactions.
• Figure 4: A guideline provided to human participants in \ref{['sec:relationship']}. Since the primary objective of the study was to assess user preferences in terms of satisfaction, no formal tutorial was provided. However, a moderator was available to offer additional explanations upon participant request. Furthermore, the information of agent models was not provided to participants.
• Figure 5: Manual categorization of factors that affect user satisfaction during interactions.