Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personality-Aware Reinforcement Learning for Persuasive Dialogue with LLM-Driven Simulation

Donghuo Zeng, Roberto Legaspi, Kazushi Ikeda

TL;DR

The paper tackles dynamic personalization in persuasive dialogue by framing interactions as a finite-horizon MDP and introducing turn-level personality estimation to condition strategy selection. It combines a Strategy-Oriented Interaction Framework with an 81-dimensional turn-level personality representation and a Dueling Double DQN (D3QN) core that uses a composite reward incorporating agreement, donation, and a change-of-mind penalty. Through LLM-driven agenda-based simulation and data augmentation on the PersuasionForGood (P4G) dataset, it demonstrates that turn-level personality conditioning improves adaptability and cumulative rewards, while simulation enhances generalization and the change-of-mind term reduces retractions. The work advances robust, interpretable, and ethically aware persuasive policies by integrating structured interaction, dynamic user modeling, and behaviorally informed rewards, with practical implications for charitable giving and social good applications. Key findings include improved policy robustness with personality signals, better generalization from LLM-driven simulation, and reduced post-agreement retractions due to the change-of-mind penalty, all under a formally defined $T=10$-turn MDP framework.

Abstract

Effective persuasive dialogue agents adapt their strategies to individual users, accounting for the evolution of their psychological states and intentions throughout conversations. We present a personality-aware reinforcement learning approach comprising three main modules: (1) a Strategy-Oriented Interaction Framework, which serves as an agenda-based strategy controller that selects strategy-level actions and generate responses via Maximal Marginal Relevance (MMR) retrieval to ensure contextual relevance, diversity, and scalable data generation; (2) Personality-Aware User Representation Learning, which produces an 81-dimensional mixed-type embedding predicted at each turn from recent exchanges and appended to the reinforcement learning state; and (3) a Dueling Double DQN (D3QN) model and Reward Prediction, in which the policy is conditioned on dialogue history and turn-level personality estimates and trained using a composite reward incorporating agreement intent, donation amount, and changeof-mind penalties. We use an agenda-based LLM simulation pipeline to generate diverse interactions, from which personality estimation is inferred from the generated utterances. Experiments on the PersuasionForGood (P4G) dataset augmented with simulated dialogues reveal three main findings: (i) turn-level personality conditioning improves policy adaptability and cumulative persuasion rewards; (ii) LLM-driven simulation enhances generalization to unseen user behaviors; and (iii) incorporating a change-of-mind penalty reduces post-agreement retractions while slightly improving donation outcomes. These results demonstrate that structured interaction, dynamic personality estimation, and behaviorally informed rewards together yield more effective persuasive policies.

Personality-Aware Reinforcement Learning for Persuasive Dialogue with LLM-Driven Simulation

TL;DR

The paper tackles dynamic personalization in persuasive dialogue by framing interactions as a finite-horizon MDP and introducing turn-level personality estimation to condition strategy selection. It combines a Strategy-Oriented Interaction Framework with an 81-dimensional turn-level personality representation and a Dueling Double DQN (D3QN) core that uses a composite reward incorporating agreement, donation, and a change-of-mind penalty. Through LLM-driven agenda-based simulation and data augmentation on the PersuasionForGood (P4G) dataset, it demonstrates that turn-level personality conditioning improves adaptability and cumulative rewards, while simulation enhances generalization and the change-of-mind term reduces retractions. The work advances robust, interpretable, and ethically aware persuasive policies by integrating structured interaction, dynamic user modeling, and behaviorally informed rewards, with practical implications for charitable giving and social good applications. Key findings include improved policy robustness with personality signals, better generalization from LLM-driven simulation, and reduced post-agreement retractions due to the change-of-mind penalty, all under a formally defined -turn MDP framework.

Abstract

Effective persuasive dialogue agents adapt their strategies to individual users, accounting for the evolution of their psychological states and intentions throughout conversations. We present a personality-aware reinforcement learning approach comprising three main modules: (1) a Strategy-Oriented Interaction Framework, which serves as an agenda-based strategy controller that selects strategy-level actions and generate responses via Maximal Marginal Relevance (MMR) retrieval to ensure contextual relevance, diversity, and scalable data generation; (2) Personality-Aware User Representation Learning, which produces an 81-dimensional mixed-type embedding predicted at each turn from recent exchanges and appended to the reinforcement learning state; and (3) a Dueling Double DQN (D3QN) model and Reward Prediction, in which the policy is conditioned on dialogue history and turn-level personality estimates and trained using a composite reward incorporating agreement intent, donation amount, and changeof-mind penalties. We use an agenda-based LLM simulation pipeline to generate diverse interactions, from which personality estimation is inferred from the generated utterances. Experiments on the PersuasionForGood (P4G) dataset augmented with simulated dialogues reveal three main findings: (i) turn-level personality conditioning improves policy adaptability and cumulative persuasion rewards; (ii) LLM-driven simulation enhances generalization to unseen user behaviors; and (iii) incorporating a change-of-mind penalty reduces post-agreement retractions while slightly improving donation outcomes. These results demonstrate that structured interaction, dynamic personality estimation, and behaviorally informed rewards together yield more effective persuasive policies.
Paper Structure (32 sections, 8 equations, 7 figures, 3 tables)

This paper contains 32 sections, 8 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Persuasive Dialogue
  4. Personalization and Personality Modeling
  5. Reinforcement Learning for Dialogue
  6. LLM-driven Simulation
  7. Our Architecture
  8. Problem Formulation
  9. Strategy-Oriented Interaction Framework
  10. Strategy Prediction from User Inputs.
  11. Strategy Selection for System Response.
  12. Utterance Retrieval and Response Construction.
  13. Personality-Aware User Representation
  14. Feature extraction and encoding.
  15. Turn-level prediction and RL augmentation.
  16. ...and 17 more sections

Figures (7)

  • Figure 1: A transition dynamics model.
  • Figure 2: Overview of the Proposed Interaction Framework
  • Figure 3: Overview of the personality-aware D3QN architecture.
  • Figure 4: Statistics of users' behavior from the P4G.
  • Figure 5: CCA correlation and marginal distributions for ground-truth and predicted psychological profiles.
  • ...and 2 more figures