Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

Hareeshwar Karthikeyan

TL;DR

The paper addresses the need for scalable, realistic testing of conversational AI by proposing a three-agent framework that decouples persona-grounded behavior from task-state tracking. By separating the User Agent, State Tracking Agent, and Message Attributes Generation Agent, the approach achieves explainability, reproducibility, and scalability in goal-directed dialogues, demonstrated in a restaurant ordering domain. Key contributions include a formal protocol for agent collaboration, a comprehensive 60-test restaurant dataset with 20 personas and 50 menu items, and a multi-metric evaluation (PAS, BVS, TRA, DEI, CRRS) showing significant improvements over single-LLM baselines. The work provides a principled blueprint for realistic user simulation across interactive domains, enabling more robust testing and evaluation of conversational systems.

Abstract

Testing conversational AI systems at scale across diverse domains necessitates realistic and diverse user interactions capturing a wide array of behavioral patterns. We present a novel multi-agent framework for realistic, explainable human user simulation in interactive scenarios, using persona control and task state tracking to mirror human cognitive processes during goal-oriented conversations. Our system employs three specialized AI agents: (1) a User Agent to orchestrate the overall interaction, (2) a State Tracking Agent to maintain structured task state, and (3) a Message Attributes Generation Agent that controls conversational attributes based on task progress and assigned persona. To validate our approach, we implement and evaluate the framework for guest ordering at a restaurant with scenarios rich in task complexity, behavioral diversity, and conversational ambiguity. Through systematic ablations, we evaluate the contributory efficacy of each agentic component to overall simulation quality in terms of persona adherence, task completion accuracy, explainability, and realism. Our experiments demonstrate that the complete multi-agent system achieves superior simulation quality compared to single-LLM baselines, with significant gains across all evaluation metrics. This framework establishes a powerful environment for orchestrating agents to simulate human users with cognitive plausibility, decomposing the simulation into specialized sub-agents that reflect distinct aspects of human thought processes applicable across interactive domains.

Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

TL;DR

The paper addresses the need for scalable, realistic testing of conversational AI by proposing a three-agent framework that decouples persona-grounded behavior from task-state tracking. By separating the User Agent, State Tracking Agent, and Message Attributes Generation Agent, the approach achieves explainability, reproducibility, and scalability in goal-directed dialogues, demonstrated in a restaurant ordering domain. Key contributions include a formal protocol for agent collaboration, a comprehensive 60-test restaurant dataset with 20 personas and 50 menu items, and a multi-metric evaluation (PAS, BVS, TRA, DEI, CRRS) showing significant improvements over single-LLM baselines. The work provides a principled blueprint for realistic user simulation across interactive domains, enabling more robust testing and evaluation of conversational systems.

Abstract

Testing conversational AI systems at scale across diverse domains necessitates realistic and diverse user interactions capturing a wide array of behavioral patterns. We present a novel multi-agent framework for realistic, explainable human user simulation in interactive scenarios, using persona control and task state tracking to mirror human cognitive processes during goal-oriented conversations. Our system employs three specialized AI agents: (1) a User Agent to orchestrate the overall interaction, (2) a State Tracking Agent to maintain structured task state, and (3) a Message Attributes Generation Agent that controls conversational attributes based on task progress and assigned persona. To validate our approach, we implement and evaluate the framework for guest ordering at a restaurant with scenarios rich in task complexity, behavioral diversity, and conversational ambiguity. Through systematic ablations, we evaluate the contributory efficacy of each agentic component to overall simulation quality in terms of persona adherence, task completion accuracy, explainability, and realism. Our experiments demonstrate that the complete multi-agent system achieves superior simulation quality compared to single-LLM baselines, with significant gains across all evaluation metrics. This framework establishes a powerful environment for orchestrating agents to simulate human users with cognitive plausibility, decomposing the simulation into specialized sub-agents that reflect distinct aspects of human thought processes applicable across interactive domains.
Paper Structure (33 sections, 15 equations, 3 figures, 3 tables)

This paper contains 33 sections, 15 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Human Simulation and Persona Modeling
  4. Multi-Agent Orchestration and Coordination
  5. Cognitive Architectures and State Management
  6. Synthetic Data Generation and Evaluation
  7. Methodology
  8. User Agent
  9. State Tracking Agent
  10. Message Attributes Generation Agent
  11. Protocol
  12. Baseline instructions
  13. Critical constraints
  14. Conversation rules
  15. Exit gating
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Comparison of simulation approaches:Left panel (Human User): Manual testing requires significant human effort including test scenario creation, multi-turn conversation management, person-in-role-playing, and result documentation, making it expensive and difficult to scale. Center panel (Single Model System): Traditional automated approaches using a single model suffer from overloaded responsibilities, attempting to simultaneously handle state tracking and memory, behavior modeling, response generation, and context management, leading to inconsistent personas and poor interpretability. Right panel (Agentic Simulation): Our proposed multi-agent framework distributes intelligence across specialized components, providing interpretable decisions, reproducible behavior, consistent personas, scalable architecture, and separated concerns, enabling systematic and reliable user simulation at scale. This decomposition mirrors human cognitive processes: tracking task completion progress (working memory) Sun2022MetaphoricalUserSimulatorsHu2025UnifiedMindModel, deciding how to respond based on personality and context (behavioral planning) Park2023GenerativeAgents, and generating appropriate utterances (language production).
  • Figure 2: Multi-agent architecture for human user simulation showing the three-agent framework: (1) User Agent serves as the primary orchestrator that generates simulated user responses by receiving input messages and invoking the two sub-agents in sequence, (2) State Tracking Agent maintains structured task state representation by tracking current confirmed items against target goals, and (3) Message Attributes Generation Agent determines behavioral characteristics (mood, execution style, exploration patterns) based on persona biography and current state.
  • Figure 3: Performance gains from baseline