Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PsychCounsel-Bench: Evaluating the Psychology Intelligence of Large Language Models

Min Zeng

TL;DR

The paper introduces PsychCounsel-Bench, a 2,252-question, exam-style benchmark derived from the U.S. National Counselor Examination to assess whether large language models can possess licensure-worthy psychological counseling knowledge. It systematically evaluates a broad set of open-source and proprietary LLMs, revealing a clear performance hierarchy where frontier models like GPT-4o exceed licensure thresholds, large open-source systems close the gap, and smaller models remain unreliable. The dataset construction combines public exam questions, GPT-based paraphrasing, and expert validation, with the dataset and code released for public use. The findings highlight both the promise of advanced LLMs in psychology-oriented tasks and the substantial work needed in domain adaptation, ethical reasoning, and safe deployment for mental health contexts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across a wide range of industries, primarily due to their impressive generative abilities. Yet, their potential in applications requiring cognitive abilities, such as psychological counseling, remains largely untapped. This paper investigates the key question: \textit{Can LLMs be effectively applied to psychological counseling?} To determine whether an LLM can effectively take on the role of a psychological counselor, the first step is to assess whether it meets the qualifications required for such a role, namely the ability to pass the U.S. National Counselor Certification Exam (NCE). This is because, just as a human counselor must pass a certification exam to practice, an LLM must demonstrate sufficient psychological knowledge to meet the standards required for such a role. To address this, we introduce PsychCounsel-Bench, a benchmark grounded in U.S.national counselor examinations, a licensure test for professional counselors that requires about 70\% accuracy to pass. PsychCounsel-Bench comprises approximately 2,252 carefully curated single-choice questions, crafted to require deep understanding and broad enough to cover various sub-disciplines of psychology. This benchmark provides a comprehensive assessment of an LLM's ability to function as a counselor. Our evaluation shows that advanced models such as GPT-4o, Llama3.3-70B, and Gemma3-27B achieve well above the passing threshold, while smaller open-source models (e.g., Qwen2.5-7B, Mistral-7B) remain far below it. These results suggest that only frontier LLMs are currently capable of meeting counseling exam standards, highlighting both the promise and the challenges of developing psychology-oriented LLMs. We release the proposed dataset for public use: https://github.com/cloversjtu/PsychCounsel-Bench

PsychCounsel-Bench: Evaluating the Psychology Intelligence of Large Language Models

TL;DR

The paper introduces PsychCounsel-Bench, a 2,252-question, exam-style benchmark derived from the U.S. National Counselor Examination to assess whether large language models can possess licensure-worthy psychological counseling knowledge. It systematically evaluates a broad set of open-source and proprietary LLMs, revealing a clear performance hierarchy where frontier models like GPT-4o exceed licensure thresholds, large open-source systems close the gap, and smaller models remain unreliable. The dataset construction combines public exam questions, GPT-based paraphrasing, and expert validation, with the dataset and code released for public use. The findings highlight both the promise of advanced LLMs in psychology-oriented tasks and the substantial work needed in domain adaptation, ethical reasoning, and safe deployment for mental health contexts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across a wide range of industries, primarily due to their impressive generative abilities. Yet, their potential in applications requiring cognitive abilities, such as psychological counseling, remains largely untapped. This paper investigates the key question: \textit{Can LLMs be effectively applied to psychological counseling?} To determine whether an LLM can effectively take on the role of a psychological counselor, the first step is to assess whether it meets the qualifications required for such a role, namely the ability to pass the U.S. National Counselor Certification Exam (NCE). This is because, just as a human counselor must pass a certification exam to practice, an LLM must demonstrate sufficient psychological knowledge to meet the standards required for such a role. To address this, we introduce PsychCounsel-Bench, a benchmark grounded in U.S.national counselor examinations, a licensure test for professional counselors that requires about 70\% accuracy to pass. PsychCounsel-Bench comprises approximately 2,252 carefully curated single-choice questions, crafted to require deep understanding and broad enough to cover various sub-disciplines of psychology. This benchmark provides a comprehensive assessment of an LLM's ability to function as a counselor. Our evaluation shows that advanced models such as GPT-4o, Llama3.3-70B, and Gemma3-27B achieve well above the passing threshold, while smaller open-source models (e.g., Qwen2.5-7B, Mistral-7B) remain far below it. These results suggest that only frontier LLMs are currently capable of meeting counseling exam standards, highlighting both the promise and the challenges of developing psychology-oriented LLMs. We release the proposed dataset for public use: https://github.com/cloversjtu/PsychCounsel-Bench

Paper Structure

This paper contains 15 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLMs in Mental Health Care
  4. LLMs in Psychological and Social Domains
  5. PsychCounsel-Bench Dataset
  6. Collecting Data
  7. Dataset Statistics
  8. Models Evaluated
  9. Task Definition
  10. Evaluation Metrics
  11. Experiments
  12. Experimental Setup
  13. Results
  14. Conclusions
  15. Limitations

Figures (1)

  • Figure 1: Overview of the PsychCounsel-Bench task, which evaluates LLMs on counselor exam-style multiple-choice items (left) using the pipeline illustrated on the right.