Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Guessing to Asking: An Approach to Resolving the Persona Knowledge Gap in LLMs during Multi-Turn Conversations

Sarvesh Baskar, Tanmay Tulsidas Verelakar, Srinivasan Parthasarathy, Manas Gaur

TL;DR

This paper addresses the persona knowledge gap in multi-turn LLM dialogues by introducing CPER, a three-module framework that dynamically detects and resolves user-specific context gaps through intrinsic uncertainty quantification and feedback-driven refinement. It formalizes uncertainty $u_t$ and knowledge gap $KG_t$, guiding clarification via a feedback loop that updates persona alignment and generates context-aware responses. Evaluations on CCPE-M (movie preferences) and ESConv (emotional support) show CPER yields higher human and GPT-based preferences and better semantic consistency than baselines, particularly in longer conversations, while exposing the inadequacy of traditional metrics like BLEU/ROUGE to capture this improvement. The work demonstrates CPER's potential to deliver more coherent, personalized, and emotionally attuned interactions, with implications for production-ready, long-horizon conversational AI systems; future work includes adaptive parameter tuning and multimodal extensions for richer context understanding.

Abstract

In multi-turn dialogues, large language models (LLM) face a critical challenge of ensuring coherence while adapting to user-specific information. This study introduces the persona knowledge gap, the discrepancy between a model's internal understanding and the knowledge required for coherent, personalized conversations. While prior research has recognized these gaps, computational methods for their identification and resolution remain underexplored. We propose Conversation Preference Elicitation and Recommendation (CPER), a novel framework that dynamically detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement. CPER consists of three key modules: a Contextual Understanding Module for preference extraction, a Dynamic Feedback Module for measuring uncertainty and refining persona alignment, and a Persona-Driven Response Generation module for adapting responses based on accumulated user context. We evaluate CPER on two real-world datasets: CCPE-M for preferential movie recommendations and ESConv for mental health support. Using A/B testing, human evaluators preferred CPER's responses 42% more often than baseline models in CCPE-M and 27% more often in ESConv. A qualitative human evaluation confirms that CPER's responses are preferred for maintaining contextual relevance and coherence, particularly in longer (12+ turn) conversations.

From Guessing to Asking: An Approach to Resolving the Persona Knowledge Gap in LLMs during Multi-Turn Conversations

TL;DR

This paper addresses the persona knowledge gap in multi-turn LLM dialogues by introducing CPER, a three-module framework that dynamically detects and resolves user-specific context gaps through intrinsic uncertainty quantification and feedback-driven refinement. It formalizes uncertainty and knowledge gap , guiding clarification via a feedback loop that updates persona alignment and generates context-aware responses. Evaluations on CCPE-M (movie preferences) and ESConv (emotional support) show CPER yields higher human and GPT-based preferences and better semantic consistency than baselines, particularly in longer conversations, while exposing the inadequacy of traditional metrics like BLEU/ROUGE to capture this improvement. The work demonstrates CPER's potential to deliver more coherent, personalized, and emotionally attuned interactions, with implications for production-ready, long-horizon conversational AI systems; future work includes adaptive parameter tuning and multimodal extensions for richer context understanding.

Abstract

In multi-turn dialogues, large language models (LLM) face a critical challenge of ensuring coherence while adapting to user-specific information. This study introduces the persona knowledge gap, the discrepancy between a model's internal understanding and the knowledge required for coherent, personalized conversations. While prior research has recognized these gaps, computational methods for their identification and resolution remain underexplored. We propose Conversation Preference Elicitation and Recommendation (CPER), a novel framework that dynamically detects and resolves persona knowledge gaps using intrinsic uncertainty quantification and feedback-driven refinement. CPER consists of three key modules: a Contextual Understanding Module for preference extraction, a Dynamic Feedback Module for measuring uncertainty and refining persona alignment, and a Persona-Driven Response Generation module for adapting responses based on accumulated user context. We evaluate CPER on two real-world datasets: CCPE-M for preferential movie recommendations and ESConv for mental health support. Using A/B testing, human evaluators preferred CPER's responses 42% more often than baseline models in CCPE-M and 27% more often in ESConv. A qualitative human evaluation confirms that CPER's responses are preferred for maintaining contextual relevance and coherence, particularly in longer (12+ turn) conversations.

Paper Structure

This paper contains 15 sections, 10 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. CPER Framework
  5. Experimental setup
  6. Experimental Results
  7. Conclusion
  8. Related works
  9. Analysis
  10. Why Do Traditional Metrics Fail?
  11. Learning from Negative Results
  12. Human Evaluation
  13. Evaluation Criteria
  14. GPT Evaluation
  15. CPER Prompts

Figures (1)

  • Figure 1: Illustration of the CPER framework applied to a user query for inspiring movie recommendations, highlighting its three key stages: context analysis, feedback processing, and persona-driven response generation. The diagram demonstrates how persona extraction, knowledge gap resolution, and iterative refinement ensure consistency and relevance. Dotted lines represent the internal process of identifying and addressing knowledge gaps.