PsychEval: A Multi-Session and Multi-Therapy Benchmark for High-Realism AI Psychological Counselor

TL;DR

PsychEval presents a pioneering benchmark for AI psychological counselors that emphasizes high realism through multi-session, multi-therapy interactions, with a unified three-stage clinical workflow and a hierarchical skill taxonomy. It combines over 2,000 client profiles and 369 real-world case reports across five therapies plus an Integrative approach, enabling rigorous memory, planning, and theory-switching capabilities. An integrated Holistic Evaluation Framework assesses both counselor proficiency and client simulation fidelity using therapy-specific and therapy-shared metrics, demonstrating superior quality and clinical fidelity relative to existing benchmarks. Finally, PsychEval is positioned as a high-fidelity reinforcement learning environment to support self-evolving, clinically responsible AI counselors, with planned enhancements in safety, ethics, and cross-domain reasoning.

Abstract

To develop a reliable AI for psychological assessment, we introduce \texttt{PsychEval}, a multi-session, multi-therapy, and highly realistic benchmark designed to address three key challenges: \textbf{1) Can we train a highly realistic AI counselor?} Realistic counseling is a longitudinal task requiring sustained memory and dynamic goal tracking. We propose a multi-session benchmark (spanning 6-10 sessions across three distinct stages) that demands critical capabilities such as memory continuity, adaptive reasoning, and longitudinal planning. The dataset is annotated with extensive professional skills, comprising over 677 meta-skills and 4577 atomic skills. \textbf{2) How to train a multi-therapy AI counselor?} While existing models often focus on a single therapy, complex cases frequently require flexible strategies among various therapies. We construct a diverse dataset covering five therapeutic modalities (Psychodynamic, Behaviorism, CBT, Humanistic Existentialist, and Postmodernist) alongside an integrative therapy with a unified three-stage clinical framework across six core psychological topics. \textbf{3) How to systematically evaluate an AI counselor?} We establish a holistic evaluation framework with 18 therapy-specific and therapy-shared metrics across Client-Level and Counselor-Level dimensions. To support this, we also construct over 2,000 diverse client profiles. Extensive experimental analysis fully validates the superior quality and clinical fidelity of our dataset. Crucially, \texttt{PsychEval} transcends static benchmarking to serve as a high-fidelity reinforcement learning environment that enables the self-evolutionary training of clinically responsible and adaptive AI counselors.

Figures (53)

• Figure 1: The Unified Flow of Psychological Counseling.
• Figure 2: The Flow of Structured Case Extraction.
• Figure 3: The skill-informed generative pipeline for multi-stage dialogue construction.
• Figure 4: Statistical information of skills and topics.
• Figure 5: An Example of Static Traits.