Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRMB: Benchmarking Reward Models in Long-Horizon CBT-based Counseling Dialogue

Yougen Zhou, Qin Chen, Ningning Zhou, Jie Zhou, Liang He

Abstract

Large language models (LLMs) hold potential for mental healthcare applications, particularly in cognitive behavioral therapy (CBT)-based counseling, where reward models play a critical role in aligning LLMs with preferred therapeutic behaviors. However, existing reward model evaluations often fail to capture alignment effectiveness in long-horizon interventions due to limited coverage of process-oriented datasets and misalignment between evaluation targets and psychological alignment objectives. To address these limitations, we present PRMB, a comprehensive benchmark tailored for evaluating reward models in multi-session CBT counseling. PRMB spans 6 sessions and 21 diverse negative scenarios, incorporating both pairwise and Best-of-N preference evaluations. We demonstrate a positive correlation between our benchmark and downstream counseling dialogue performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art reward models, revealing their generalization defects that were not discovered by previous benchmarks and highlighting the potential of generative reward models. Furthermore, we delve into examining the effectiveness of inference-time strategy for the evaluation of reward models and analyzing the impact factors of generative reward models. This work advances intelligent informatics for personalized healthcare by establishing a framework for reward model assessment in mental health dialogues. Evaluation code and datasets are publicly available at https://github.com/YouKenChaw/PRMB

PRMB: Benchmarking Reward Models in Long-Horizon CBT-based Counseling Dialogue

Abstract

Large language models (LLMs) hold potential for mental healthcare applications, particularly in cognitive behavioral therapy (CBT)-based counseling, where reward models play a critical role in aligning LLMs with preferred therapeutic behaviors. However, existing reward model evaluations often fail to capture alignment effectiveness in long-horizon interventions due to limited coverage of process-oriented datasets and misalignment between evaluation targets and psychological alignment objectives. To address these limitations, we present PRMB, a comprehensive benchmark tailored for evaluating reward models in multi-session CBT counseling. PRMB spans 6 sessions and 21 diverse negative scenarios, incorporating both pairwise and Best-of-N preference evaluations. We demonstrate a positive correlation between our benchmark and downstream counseling dialogue performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art reward models, revealing their generalization defects that were not discovered by previous benchmarks and highlighting the potential of generative reward models. Furthermore, we delve into examining the effectiveness of inference-time strategy for the evaluation of reward models and analyzing the impact factors of generative reward models. This work advances intelligent informatics for personalized healthcare by establishing a framework for reward model assessment in mental health dialogues. Evaluation code and datasets are publicly available at https://github.com/YouKenChaw/PRMB
Paper Structure (38 sections, 2 equations, 11 figures, 6 tables)

This paper contains 38 sections, 2 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Simulated Psychological Dialogue
  4. Reward Model
  5. Benchmarks of Reward Models
  6. Data Construction
  7. Prompt-Response Construction
  8. Counseling Case Sourcing
  9. Prompt Design
  10. Response Generation
  11. Response Filtering
  12. Partial Order Construction
  13. Pairwise Preference Construction
  14. Agreement with Human Preference
  15. Best-of-N Set Construction
  16. ...and 23 more sections

Figures (11)

  • Figure 1: CBT-based counseling dialogues require session-level coherence, long-term consistency across multiple sessions, and adherence to structured therapeutic progression.
  • Figure 2: An overview of data construction process: (1) Sampling multi-session dialogues into prompts and obtaining multiple responses for them. (2) Organizing them into pairs or best-of-N lists.
  • Figure 3: Distribution of prompts (with percentages annotated above each bar), pairwise preference pairs, and Best-of-N queries across therapeutic sessions. Pairwise and BoN counts are shown inside the respective stacked segments. Session themes: (1) Information gathering, goal setting, and case conceptualization; (2) Identifying and challenging automated thoughts; (3) Discussion case conceptualization; (4) Adjusting intermediate beliefs and core beliefs; (5) Relapse prevention; (6) Consolidation.
  • Figure 4: Percentage distribution of negative experience categories in the pairwise and BoN loser sets, grouped by the four-cluster taxonomy.
  • Figure 5: (a) Average accuracy (pairwise + BoN), (b) Pairwise accuracy, and (c) Best-of-N accuracy across the six therapeutic sessions. Generative LLM-as-a-judge models (dashed lines) and discriminative reward models (solid lines) are distinguished by color and marker style. Solid lines represent discriminative reward models; dashed lines represent generative LLM-as-a-judge models.
  • ...and 6 more figures