PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Models

TL;DR

PsyEval introduces a comprehensive, task-driven benchmark suite to evaluate large language models in mental health across knowledge, diagnostic, and emotional-support dimensions. It combines five sub-tasks over three categories, using diverse datasets (USMLE-mental, Crisis Response QA, SMHD, D4, PsyQA) and a mix of automatic and human evaluation metrics to probe knowledge, diagnostic reasoning, empathy, and safety. Key findings show GPT-4 leads in knowledge QA but struggles with nuanced diagnostics and empathetic counseling, while fine-tuned models offer mixed gains and context-window constraints shape performance. The work highlights significant room for improvement in LLMs' ethical, safe, and empathetic handling of mental-health content, and provides a structured path for future model optimization and evaluation. PsyEval thus sets a targeted benchmark standard for capturing the specialized challenges of mental-health related NLP systems.

Abstract

Evaluating Large Language Models (LLMs) in the mental health domain poses distinct challenged from other domains, given the subtle and highly subjective nature of symptoms that exhibit significant variability among individuals. This paper presents PsyEval, the first comprehensive suite of mental health-related tasks for evaluating LLMs. PsyEval encompasses five sub-tasks that evaluate three critical dimensions of mental health. This comprehensive framework is designed to thoroughly assess the unique challenges and intricacies of mental health-related tasks, making PsyEval a highly specialized and valuable tool for evaluating LLM performance in this domain. We evaluate twelve advanced LLMs using PsyEval. Experiment results not only demonstrate significant room for improvement in current LLMs concerning mental health but also unveil potential directions for future model optimization.

Figures (14)

• Figure 1: Overview diagram of PsyEval.
• Figure 2: Data Collection Steps of Crisis Response QA
• Figure 3: Example for Mental Health QA
• Figure 4: Analysis of the evaluated LLMs on PsyEval.a Comparison of model performance on the USMLE-mental and Crisis Response QA datasets, and comparison of model performance on two different types of questions within the USMLE-mental dataset. b Models' performance on the Diagnosis Prediction via Online Text Data task. c Models' performance on the Diagnosis Prediction via Dialogue task. d Models' performance on the Psychological Counseling task, assessed by human evaluators. e Models' performance in terms of empathy on the Psychological Counseling task, assessed by human evaluators.
• Figure 5: Prompt for Mental health Question-Answering