When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation

TL;DR

This paper tackles the challenge of evaluating LLMs in mental health by introducing MentalBench-100k, a large-scale, authentic-dialogue benchmark with multi-LLM and human responses, and MentalAlign-70k, a dual-axis CSS/ARS evaluation framework assessed by both four LLM judges and clinical experts. It then formalizes the Affective–Cognitive Agreement Framework using two-way mixed-effects ICC with bootstrap CIs to quantify human–LLM judge reliability, bias, and agreement across seven therapeutic attributes. The key finding is that cognitive attributes exhibit strong reliability for ranking (ICC(C,1) up to ~0.95) and varying absolute agreement, while affective and safety-related attributes show inflation, wide CIs, and limited absolute agreement, signaling the need for calibrated or human-involved evaluation in those dimensions. The authors provide open-source benchmarks and guidelines for when automated evaluation is trustworthy versus when human oversight is essential, advancing safer and more scalable AI deployment in mental health contexts.

Abstract

Evaluating Large Language Models (LLMs) for mental health support is challenging due to the emotionally and cognitively complex nature of therapeutic dialogue. Existing benchmarks are limited in scale, reliability, often relying on synthetic or social media data, and lack frameworks to assess when automated judges can be trusted. To address the need for large-scale dialogue datasets and judge reliability assessment, we introduce two benchmarks that provide a framework for generation and evaluation. MentalBench-100k consolidates 10,000 one-turn conversations from three real scenarios datasets, each paired with nine LLM-generated responses, yielding 100,000 response pairs. MentalAlign-70k}reframes evaluation by comparing four high-performing LLM judges with human experts across 70,000 ratings on seven attributes, grouped into Cognitive Support Score (CSS) and Affective Resonance Score (ARS). We then employ the Affective Cognitive Agreement Framework, a statistical methodology using intraclass correlation coefficients (ICC) with confidence intervals to quantify agreement, consistency, and bias between LLM judges and human experts. Our analysis reveals systematic inflation by LLM judges, strong reliability for cognitive attributes such as guidance and informativeness, reduced precision for empathy, and some unreliability in safety and relevance. Our contributions establish new methodological and empirical foundations for reliable, large-scale evaluation of LLMs in mental health. We release the benchmarks and codes at: https://github.com/abeerbadawi/MentalBench/

Figures (5)

• Figure 1: Overview of our proposed system: MentalBench-100k provides mental health conversations with multi-LLM responses. MentalAlign-70k benchmarks cognitive and affective attributes using human experts and LLMs as judges. Affective–Cognitive Agreement framework applies ICC and bias detection to quantify reliability.
• Figure 2: Precision–reliability patterns by judge and attribute. Left: ICC(C,1) heatmap. Right: CI-width heatmap. Columns are ordered cognitive $\rightarrow$ affective $\rightarrow$ safety/relevance to expose the domain split.
• Figure 3: Top 15 most common combinations of up to three mental health condition labels assigned per conversation from the Dataset.
• Figure 4: Comparison of human baseline ratings with four LLM judges (Claude-3.7-Sonnet, GPT-4o, O4-Mini, and Gemini-2.5-Flash) across nine models. Each bar represents the average evaluation score (1–5) over 1,000 conversations, aggregated across all seven attributes.
• Figure 5: Diagnostic power comparison: Traditional metrics vs. ICC methodology with bootstrap confidence intervals. Scenario A shows how MSE misclassifies systematic bias as incompetence, while ICC enables calibration of an excellent judge. Scenario B demonstrates how point estimates mask uncertainty that bootstrap analysis reveals. Both scenarios illustrate critical reliability decisions that traditional metrics would handle incorrectly.