CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy

TL;DR

CBT-Bench proposes a hierarchical benchmark to evaluate LLM-assisted cognitive-behavioral therapy across three levels: knowledge recall (CBT-QA), cognitive-model understanding (CBT-CD/CBT-PC/CBT-FC), and therapeutic response generation (CBT-DP). Across six LLMs, results show strong performance on factual CBT knowledge but limited capacity for nuanced cognitive modeling and patient-centered therapeutic dialogue, underscoring the gap between AI capabilities and real-world psychotherapy needs. The study highlights the necessity of expert oversight, ethical safeguards, and continued dataset refinement to responsibly augment therapists rather than replace them. Overall, CBT-Bench provides a principled framework and valuable insights to guide future research on AI-assisted mental health care.

Abstract

There is a significant gap between patient needs and available mental health support today. In this paper, we aim to thoroughly examine the potential of using Large Language Models (LLMs) to assist professional psychotherapy. To this end, we propose a new benchmark, CBT-BENCH, for the systematic evaluation of cognitive behavioral therapy (CBT) assistance. We include three levels of tasks in CBT-BENCH: I: Basic CBT knowledge acquisition, with the task of multiple-choice questions; II: Cognitive model understanding, with the tasks of cognitive distortion classification, primary core belief classification, and fine-grained core belief classification; III: Therapeutic response generation, with the task of generating responses to patient speech in CBT therapy sessions. These tasks encompass key aspects of CBT that could potentially be enhanced through AI assistance, while also outlining a hierarchy of capability requirements, ranging from basic knowledge recitation to engaging in real therapeutic conversations. We evaluated representative LLMs on our benchmark. Experimental results indicate that while LLMs perform well in reciting CBT knowledge, they fall short in complex real-world scenarios requiring deep analysis of patients' cognitive structures and generating effective responses, suggesting potential future work.

Figures (12)

• Figure 1: Detailed F1 scores of each label for CBT-CD and CBT-FC.
• Figure 2: The overall pairwise comparison of different models vs. reference across difficulty level.
• Figure 3: An example cognitive model from beck2020cognitive.
• Figure 4: Detailed accuracies on different types of knowledge for CBT-QA and the F1 score of each label for CBT-PC.
• Figure 5: The win-tie-loss comparison among different models on three difficulty levels.