Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback

Shijing Zhu, Zhuang Chen, Guanqun Bi, Binghang Li, Yaxi Deng, Dazhen Wan, Libiao Peng, Xiyao Xiao, Rongsheng Zhang, Tangjie Lv, Zhipeng Hu, FangFang Li, Minlie Huang

TL;DR

Ψ-Arena introduces a tripartite, closed-loop framework for evaluating and optimizing LLM-based psychological counselors using interactive NPC clients and multi-source feedback from clients, supervisors, and counselors. The framework combines realistic, multi-stage counseling simulations with three evaluation scales and an automated feedback loop that leverages established counseling knowledge to iteratively improve model responses. Across eight LLMs, the study reveals substantive performance disparities by perspective, demonstrates strong alignment with human experts in many cases, and reports up to a 141% relative improvement after optimization. The work highlights the importance of multi-perspective assessment, feedback-driven refinement, and careful consideration of ethics, privacy, and cultural bias in deploying AI-assisted mental health tools at scale.

Abstract

Large language models (LLMs) have shown promise in providing scalable mental health support, while evaluating their counseling capability remains crucial to ensure both efficacy and safety. Existing evaluations are limited by the static assessment that focuses on knowledge tests, the single perspective that centers on user experience, and the open-loop framework that lacks actionable feedback. To address these issues, we propose Ψ-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients, (2) Tripartite evaluation that integrates assessments from the client, counselor, and supervisor perspectives, and (3) Closed-loop optimization that iteratively improves LLM counselors using diagnostic feedback. Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives. Moreover, reflection-based optimization results in up to a 141% improvement in counseling performance. We hope PsychoArena provides a foundational resource for advancing reliable and human-aligned LLM applications in mental healthcare.

Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback

TL;DR

Ψ-Arena introduces a tripartite, closed-loop framework for evaluating and optimizing LLM-based psychological counselors using interactive NPC clients and multi-source feedback from clients, supervisors, and counselors. The framework combines realistic, multi-stage counseling simulations with three evaluation scales and an automated feedback loop that leverages established counseling knowledge to iteratively improve model responses. Across eight LLMs, the study reveals substantive performance disparities by perspective, demonstrates strong alignment with human experts in many cases, and reports up to a 141% relative improvement after optimization. The work highlights the importance of multi-perspective assessment, feedback-driven refinement, and careful consideration of ethics, privacy, and cultural bias in deploying AI-assisted mental health tools at scale.

Abstract

Large language models (LLMs) have shown promise in providing scalable mental health support, while evaluating their counseling capability remains crucial to ensure both efficacy and safety. Existing evaluations are limited by the static assessment that focuses on knowledge tests, the single perspective that centers on user experience, and the open-loop framework that lacks actionable feedback. To address these issues, we propose Ψ-Arena, an interactive framework for comprehensive assessment and optimization of LLM-based counselors, featuring three key characteristics: (1) Realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients, (2) Tripartite evaluation that integrates assessments from the client, counselor, and supervisor perspectives, and (3) Closed-loop optimization that iteratively improves LLM counselors using diagnostic feedback. Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives. Moreover, reflection-based optimization results in up to a 141% improvement in counseling performance. We hope PsychoArena provides a foundational resource for advancing reliable and human-aligned LLM applications in mental healthcare.
Paper Structure (47 sections, 5 figures, 24 tables)

This paper contains 47 sections, 5 figures, 24 tables.

Table of Contents

  1. Introduction
  2. -Arena
  3. Framework Overview
  4. Client in -Arena
  5. Client Profiles
  6. Client Behaviors
  7. Trust-Building Phase
  8. Diagnostic Phase
  9. Solution-Exploration Phase
  10. Client Simulation
  11. Counselor in -Arena
  12. Tripartite Evaluation Metrics
  13. Client-Oriented Scale
  14. Supervisor-Oriented Scale
  15. Counselor-Oriented Scale
  16. ...and 32 more sections

Figures (5)

  • Figure 1: The comparison between -Arena and existing studies on evaluating LLM-base counselors.
  • Figure 2: In -Arena, LLM-based counselors interact with NPC clients, receive multi-source evaluations, and improve counseling performance through self-reflection.
  • Figure 3: Comparison of consistency results between -Arena and human experts.
  • Figure 4: Comparison of model pass rates before and after optimization.
  • Figure 5: Fine-grained counseling performance of all LLMs on different topics and dimensions.