Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planning

TL;DR

This work addresses the neglect of user-specific characteristics in dialogue policy planning by proposing UDP, a three-stage framework that builds an intrinsic User World Model. It combines a diffusion-based User Persona Portrayer, a Brownian Bridge–inspired User Feedback Anticipator, and a Transformer-based User-Aware Policy Planner, enhanced by active online learning. The authors validate UDP through task-specific personas on P4G and ESConv, showing robust improvements in user-tailored policy planning and better handling of diverse user traits. The approach offers a human-centric, adaptable pathway for personalized dialogue systems with demonstrated generalizability and practical impact.

Abstract

Recent advancements in dialogue policy planning have emphasized optimizing system agent policies to achieve predefined goals, focusing on strategy design, trajectory acquisition, and efficient training paradigms. However, these approaches often overlook the critical role of user characteristics, which are essential in real-world scenarios like conversational search and recommendation, where interactions must adapt to individual user traits such as personality, preferences, and goals. To address this gap, we first conduct a comprehensive study utilizing task-specific user personas to systematically assess dialogue policy planning under diverse user behaviors. By leveraging realistic user profiles for different tasks, our study reveals significant limitations in existing approaches, highlighting the need for user-tailored dialogue policy planning. Building on this foundation, we present the User-Tailored Dialogue Policy Planning (UDP) framework, which incorporates an Intrinsic User World Model to model user traits and feedback. UDP operates in three stages: (1) User Persona Portraying, using a diffusion model to dynamically infer user profiles; (2) User Feedback Anticipating, leveraging a Brownian Bridge-inspired anticipator to predict user reactions; and (3) User-Tailored Policy Planning, integrating these insights to optimize response strategies. To ensure robust performance, we further propose an active learning approach that prioritizes challenging user personas during training. Comprehensive experiments on benchmarks, including collaborative and non-collaborative settings, demonstrate the effectiveness of UDP in learning user-specific dialogue strategies. Results validate the protocol's utility and highlight UDP's robustness, adaptability, and potential to advance user-centric dialogue systems.

Figures (6)

• Figure 1: Framework overview. The framework UDP consists of three stages: User Persona Portraying, User Feedback Anticipating, and User-Tailored Policy Planning.
• Figure 2: The portraying (denoising) process of the Diffusion Model-based User Persona Portraying stage.
• Figure 3: The architecture of the Brownian Bridge Process-based User Feedback Anticipating stage.
• Figure 4: The architecture comparison of the Policy Planning stage: (a) PPDPP & TRIP; (b) UDP (Our).
• Figure 5: The dialogue agents' performance across various personas on P4G and ESConv. UDP achieves improvements on almost all personas.