Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward

TL;DR

CURIO introduces a curiosity-driven, user-modeling intrinsic reward to online, multi-turn RLHF personalization. By treating the user as the environment in a POMDP and maintaining a belief distribution over user types, CURIO uses per-turn intrinsic rewards derived from belief improvements to actively learn about the user while pursuing end-task goals. The approach is connected to potential-based reward shaping to preserve optimal policies while boosting sample efficiency. Empirical results in Exercise Recommendation and Education Dialogue show CURIO improves personalization and generalization without sacrificing conversation quality, with supportive human evaluations and analyses of reward-hacking risks. This framework enables real-time, domain-agnostic personalization in dialogue systems, with potential impact in education and healthcare domains.

Abstract

Effective conversational agents like large language models (LLMs) must personalize their interactions to adapt to user preferences, personalities, and attributes across diverse domains like education and healthcare. Current methods like Reinforcement Learning from Human Feedback (RLHF), often prioritize helpfulness and safety but fall short in fostering truly empathetic, adaptive, and personalized dialogues. Existing personalization approaches typically rely on extensive user history, limiting their effectiveness for new or context-limited users. To address these limitations, we propose leveraging a user model to incorporate a curiosity-based intrinsic reward into multi-turn RLHF. This novel reward mechanism encourages the LLM agent to actively infer user traits by optimizing conversations to improve its user model's accuracy. Consequently, the agent delivers more personalized interactions by learning more about the user. We demonstrate our method's effectiveness in two distinct domains: significantly improving personalization performance in a conversational recommendation task, and personalizing conversations for different learning styles in an educational setting. We show improved generalization capabilities compared to traditional multi-turn RLHF, all while maintaining conversation quality. Our method offers a promising solution for creating more personalized, adaptive, and engaging conversational agents.

Figures (5)

• Figure 1: Our work focuses on training personalized LLMs in multi-turn conversations. Standard LLM training methods treat all users as a homogeneous group, leading to suboptimal performance for different groups (top left); while an optimal LLM can actively learn about user preferences within the conversation and then adapt to it (top right). We introduce Intrinsic Motivation in user modeling to multi-turn RLHF. Intuitively, rather than training an LLM only with the end-of-conversation sparse reward, we add an additional turn-based reward that is given by its improvement in belief over the user type after generating an utterance and receiving a response, which guides the LLM to actively learn about user type and then adapt to each user throughout the conversation.
• Figure 2: RL fine-tuning Pipeline for CURIO framework in one episode. We leverage a user model to obtain dense turn-based intrinsic rewards as a supplement to the sparse end-of-conversation rewards.
• Figure 3: Training curves for Exercise Recommendation.
• Figure 4: Calibrated user modeling accuracy for baseline vs CURIO model (DiffAcc) at the third turn in Education Dialogue. y-axis: $b(u^*)-1/2$. The conversations generated with our model consistently reveal more user information.
• Figure 5: Questions distribution of CURIO model, RLHF baseline, and SFT initial checkpoint. The left ones are relevant attributes, and the right ones are irrelevant to the strategy recommendation.