Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

Jing Ye, Xinpei Zhao, Lu Xiang, Yaping Zhang, Chengqing Zong

Abstract

While current emotional support dialogue systems typically rely on expert-defined scalar rewards for alignment, these signals suffer from severe information sparsity. They cannot explain why a response failed or how to adapt to dynamic user states, often diverging from the actual goal of facilitating positive emotional shifts. In practice, the most direct and reliable learning signal emerges from the user's continuous reactions during ongoing interaction. We therefore propose Reaction Aware Policy Optimization (RAPO), a framework that optimizes over interaction consequences rather than rubric scores. RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback through three core components: Hindsight Dialogue Selection, which isolates pivotal turns that meaningfully alter user emotional trajectories; Generative Hindsight Feedback, which transforms user reactions into contrastive ranking signals and natural-language critiques; and Scalar-Verbal Hybrid Policy Optimization, which couples scalar reward optimization for global alignment with verbal feedback distillation for fine-grained semantic refinement. Extensive experiments on ESC and Sotopia demonstrate that RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.

Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learning

Abstract

While current emotional support dialogue systems typically rely on expert-defined scalar rewards for alignment, these signals suffer from severe information sparsity. They cannot explain why a response failed or how to adapt to dynamic user states, often diverging from the actual goal of facilitating positive emotional shifts. In practice, the most direct and reliable learning signal emerges from the user's continuous reactions during ongoing interaction. We therefore propose Reaction Aware Policy Optimization (RAPO), a framework that optimizes over interaction consequences rather than rubric scores. RAPO treats dialogue as a reaction-driven process and utilizes simulated user responses to generate dense natural-language feedback through three core components: Hindsight Dialogue Selection, which isolates pivotal turns that meaningfully alter user emotional trajectories; Generative Hindsight Feedback, which transforms user reactions into contrastive ranking signals and natural-language critiques; and Scalar-Verbal Hybrid Policy Optimization, which couples scalar reward optimization for global alignment with verbal feedback distillation for fine-grained semantic refinement. Extensive experiments on ESC and Sotopia demonstrate that RAPO significantly outperforms strong reinforcement learning baselines in driving positive interaction outcomes.
Paper Structure (69 sections, 23 equations, 4 figures, 9 tables)

This paper contains 69 sections, 23 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Group Relative Policy Optimization
  4. On-Policy Distillation
  5. Method
  6. Reaction-Aware Problem Formulation
  7. Hindsight Dialogue Selection
  8. Generative Hindsight Feedback
  9. Group-wise Exploration and User Simulation.
  10. Contrastive Critique Generation.
  11. Scalar–Verbal Hybrid Policy Optimization
  12. Scalar RL
  13. Verbal RL
  14. Self-Teacher Construction.
  15. Optimization.
  16. ...and 54 more sections

Figures (4)

  • Figure 1: Comparison between expert-centric scalar rewards and user-reaction aware mixed rewards.
  • Figure 2: Overview of the RAPO framework. It integrates user simulation, contrastive critique generation, and scalar-verbal hybrid policy optimization.
  • Figure 3: Results of the pair-wise human evaluation. We randomly sampled 50 simulated dialogue instances for each task. For each instance, annotators are presented with the dialogue context and two candidate responses (A and B) generated by different models. $\blacksquare$ indicates 'A win', $\blacksquare$ indicates 'tie', and $\blacksquare$ indicates 'B win'.
  • Figure 4: Training dynamics on ESConv. Left: Actor policy entropy during training. Middle: Average critic reward score. Right: Mean response length generated by the policy.