SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation

TL;DR

SAGE tackles the evaluation bottleneck for multi-turn agents by introducing a knowledge-grounded user simulator that merges top-down business logic (ICP-based attributes) with bottom-up agent infrastructure knowledge (product catalogs, FAQs, knowledge bases). The framework proceeds through configure-knowledge, build-task-scenarios, generate-interactions, and evaluate-id-bugs, leveraging LLMs for goal generation and a two-stage bug identification workflow (LLM-as-a-Judge plus human validation) to surface concrete agent errors. Empirical results show that SAGE yields more human-like, diverse interactions and identifies more bugs than baselines across two use cases and multiple LLMs, supporting its utility for bug-finding and iterative agent improvement. The authors also discuss limitations, such as focus on customer-facing attributes and single-goal scenarios, and propose future work on multi-goal sessions and real-world log evaluation to enhance realism and generalizability.

Abstract

Evaluating multi-turn interactive agents is challenging due to the need for human assessment. Evaluation with simulated users has been introduced as an alternative, however existing approaches typically model generic users and overlook the domain-specific principles required to capture realistic behavior. We propose SAGE, a novel user Simulation framework for multi-turn AGent Evaluation that integrates knowledge from business contexts. SAGE incorporates top-down knowledge rooted in business logic, such as ideal customer profiles, grounding user behavior in realistic customer personas. We further integrate bottom-up knowledge taken from business agent infrastructure (e.g., product catalogs, FAQs, and knowledge bases), allowing the simulator to generate interactions that reflect users' information needs and expectations in a company's target market. Through empirical evaluation, we find that this approach produces interactions that are more realistic and diverse, while also identifying up to 33% more agent errors, highlighting its effectiveness as an evaluation tool to support bug-finding and iterative agent improvement.

Figures (4)

• Figure 1: An overview of SAGE, which consists of four processes: configure knowledge, build task scenarios, generate interactions, and evaluate agent. The knowledge configuration step defines the top-down (user attributes) and bottom-up (agent infrastructure knowledge) knowledge. This knowledge is leveraged to construct task scenarios which guide the user simulator in its interactions with the agent being evaluated. The resulting interactions are analyzed to identify errors in the agent's response. The complete user profile is shown in Appendix \ref{['sec:app_profile']} and the interaction examples are shown in Table \ref{['tab:error_examples']}.
• Figure 2: Number of unique errors across high-level error categories for the RAG-based Sales Agent under w/o ICP, w/o Agent-Infra, and SAGE settings.
• Figure 3: A user profile example
• Figure 4: The number of unique errors in each category, grouped by their occurrence in the first turn versus subsequent turns.