PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments

TL;DR

PsychePass tackles the core challenge of evaluating LLMs in therapy by eliminating unanchored drift through two traceable trajectories: scripted interaction trajectories in simulated clients and dynamic, pairwise battle trajectories to derive robust Elo rankings. The framework uses a four-round Swiss-system tournament and the Bradley-Terry Elo model to produce stable, discriminative assessments across 12 competency dimensions, with a trajectory-based RL loop that translates tournament signals into policy improvements. Empirical results show strong alignment with human experts (high Cohen's kappa) and demonstrate that RL-aligned models improve in key dimensions like Diversity and Empathy, while highlighting nuanced trade-offs in others. The work provides a scalable, interpretable calibration pipeline and a path toward improving AI-assisted counseling within clearly defined boundaries, while acknowledging multimodal and real-world complexity beyond its current text-based scope.

Abstract

While large language models show promise in mental healthcare, evaluating their therapeutic competence remains challenging due to the unstructured and longitudinal nature of counseling. We argue that current evaluation paradigms suffer from an unanchored defect, leading to two forms of instability: process drift, where unsteered client simulation wanders away from specific counseling goals, and standard drift, where static pointwise scoring lacks the stability for reliable judgment. To address this, we introduce Ps, a unified framework that calibrates the therapeutic competence of LLMs via trajectory-anchored tournaments. We first anchor the interaction trajectory in simulation, where clients precisely control the fluid consultation process to probe multifaceted capabilities. We then anchor the battle trajectory in judgments through an efficient Swiss-system tournament, utilizing dynamic pairwise battles to yield robust Elo ratings. Beyond ranking, we demonstrate that tournament trajectories can be transformed into credible reward signals, enabling on-policy reinforcement learning to enhance LLMs' performance. Extensive experiments validate the effectiveness of PsychePass and its strong consistency with human expert judgments.

Figures (9)

• Figure 1: PsychePass anchors two traceable trajectories for calibrating LLMs' therapeutic competence.
• Figure 2: The overall framework of PsychePass.
• Figure 3: The win rates of the base model and the aligned model across 12 dimensions.
• Figure 4: Convergence of Elo ratings.
• Figure 5: Position bias before and after debiasing.