Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning

TL;DR

The paper addresses subjectivity in user satisfaction by modeling both individual-specific and group-level preferences (majority vs. minority) in dialogue systems. It introduces CoPeR for interpretable, user-specific reasoning, and M2PC, an EM-based perplexity routing approach that discovers latent user groups unsupervisedly. These components feed into PAda-PPO, a preference-adaptive reinforcement learning framework that jointly optimizes for individual and group satisfaction signals. Empirical results on the ESConv dataset show consistent improvements in satisfaction estimation, particularly for underrepresented groups, while highlighting limitations related to synthesized reasoning quality and subgroup coverage. Overall, the work advances nuanced alignment by balancing personalization with group diversity, supporting more inclusive conversational AI deployment.

Abstract

User satisfaction in dialogue systems is inherently subjective. When the same response strategy is applied across users, minority users may assign different satisfaction ratings than majority users due to variations in individual intents and preferences. However, existing alignment methods typically train one-size-fits-all models that aim for broad consensus, often overlooking minority perspectives and user-specific adaptation. We propose a unified framework that models both individual- and group-level preferences for user satisfaction estimation. First, we introduce Chain-of-Personalized-Reasoning (CoPeR) to capture individual preferences through interpretable reasoning chains. Second, we propose an expectation-maximization-based Majority-Minority Preference-Aware Clustering (M2PC) algorithm that discovers distinct user groups in an unsupervised manner to learn group-level preferences. Finally, we integrate these components into a preference-adaptive reinforcement learning framework (PAda-PPO) that jointly optimizes alignment with both individual and group preferences. Experiments on the Emotional Support Conversation dataset demonstrate consistent improvements in user satisfaction estimation, particularly for underrepresented user groups.

Figures (8)

• Figure 1: Majority and minority users may assign different satisfaction ratings to system responses employing the same strategy due to varying individual intents and preferences. Additionally, users within the same group may exhibit similar preference patterns (e.g., cognition-oriented versus emotion-oriented strategies). This suggests that modeling both individual-specific and group-level preferences could be an effective approach for evaluating user feedback in dialogue systems.
• Figure 2: The architecture of our proposed method for the user's satisfaction $y$ estimation, $x$ is the concatenation of context $c$ and \ref{['sec:UCOT_prompts']}$r_\text{ucot}$.
• Figure 3: Distribution of feedback scores for cognition‑ and emotion‑oriented supporter response strategies across majority and minority user groups from ESConv dataset.
• Figure 4: Results on the EM iterations during the Majority-Minority Preference-Aware Clustering stage.
• Figure 5: Base prompt.