Preference Learning Unlocks LLMs' Psycho-Counseling Skills

TL;DR

This work tackles the data and alignment bottlenecks hindering LLM-based psycho-counseling by creating PsychoCounsel-Preference, a large therapist-aligned preference resource generated from 26k client speeches and 36k+ preference pairs evaluated against seven professional principles. It advances reward modeling and online preference learning to train PsychoCounsel-Policy agents, achieving state-of-the-art performance with PsychoCounsel-Llama3-8B that beats GPT-4o by substantial margins (87.0% win rate without length constraints). The approach emphasizes therapist-centered evaluation, diversity of client topics, and rigorous validation, including expert agreement on annotations, and it releases the datasets and models to enable further research. The work demonstrates that online preference data and model scale contribute to higher-quality, safer, and more autonomous-looking psycho-counseling behaviors while maintaining essential human oversight and ethical safeguards.

Abstract

Applying large language models (LLMs) to assist in psycho-counseling is an emerging and meaningful approach, driven by the significant gap between patient needs and the availability of mental health support. However, current LLMs struggle to consistently provide effective responses to client speeches, largely due to the lack of supervision from high-quality real psycho-counseling data, whose content is typically inaccessible due to client privacy concerns. Furthermore, the quality of therapists' responses in available sessions can vary significantly based on their professional training and experience. Assessing the quality of therapists' responses remains an open challenge. In this work, we address these challenges by first proposing a set of professional and comprehensive principles to evaluate therapists' responses to client speeches. Using these principles, we create a preference dataset, PsychoCounsel-Preference, which contains 36k high-quality preference comparison pairs. This dataset aligns with the preferences of professional psychotherapists, providing a robust foundation for evaluating and improving LLMs in psycho-counseling. Experiments on reward modeling and preference learning demonstrate that PsychoCounsel-Preference is an excellent resource for LLMs to acquire essential skills for responding to clients in a counseling session. Our best-aligned model, PsychoCounsel-Llama3-8B, achieves an impressive win rate of 87% against GPT-4o. We release PsychoCounsel-Preference, PsychoCounsel-Llama3-8B and the reward model PsychoCounsel Llama3-8B-Reward to facilitate the research of psycho-counseling with LLMs at: https://hf.co/Psychotherapy-LLM.

Figures (6)

• Figure 1: PsychoCounsel-Preference Construction Pipeline. 1) We first collect over 26k client speeches covering a wide range of topics from various sources, applying necessary data cleaning. 2) 20 popular LLMs are sampled and prompted to roleplay as psychotherapists and give responses to these client speeches. 3) GPT-4o is instructed to evaluate the responses based on our proposed PsychoCounsel Principles, and preference pairs with substantial score gaps are incorporated into PsychoCounsel-Preference.
• Figure 2: Experts' Comparison between GPT-4o and PychoChat-Llama3-8B in Two Settings
• Figure 3: Absolute Scores
• Figure 4: Comparison of Training Online or Offline
• Figure 5: Chosen Model Distribution