MindEval: Benchmarking Language Models on Multi-turn Mental Health Support

TL;DR

MindEval tackles the lack of realistic benchmarks for multi-turn AI-based mental health support by introducing an automatic, model-agnostic framework built with clinical expert input. It combines patient simulations with judge-based automatic evaluation and validates both realism and scoring against human judgments. Benchmarking 12 state-of-the-art systems reveals widespread underperformance, especially in AI-specific communication quality, with reasoning ability and scale not guaranteeing better outcomes and performance dropping for longer or more severe interactions. The work provides a reproducible platform, released data and prompts, and sets a foundation for future improvements in safe, effective AI-assisted therapy.

Abstract

Demand for mental health support through AI chatbots is surging, though current systems present several limitations, like sycophancy or overvalidation, and reinforcement of maladaptive beliefs. A core obstacle to the creation of better systems is the scarcity of benchmarks that capture the complexity of real therapeutic interactions. Most existing benchmarks either only test clinical knowledge through multiple-choice questions or assess single responses in isolation. To bridge this gap, we present MindEval, a framework designed in collaboration with Ph.D-level Licensed Clinical Psychologists for automatically evaluating language models in realistic, multi-turn mental health therapy conversations. Through patient simulation and automatic evaluation with LLMs, our framework balances resistance to gaming with reproducibility via its fully automated, model-agnostic design. We begin by quantitatively validating the realism of our simulated patients against human-generated text and by demonstrating strong correlations between automatic and human expert judgments. Then, we evaluate 12 state-of-the-art LLMs and show that all models struggle, scoring below 4 out of 6, on average, with particular weaknesses in problematic AI-specific patterns of communication. Notably, reasoning capabilities and model scale do not guarantee better performance, and systems deteriorate with longer interactions or when supporting patients with severe symptoms. We release all code, prompts, and human evaluation data.

Figures (14)

• Figure 1: The MindEval framework for evaluating a clinician LLM in mental health therapy interactions.
• Figure 2: t-SNE visualization of user response Gemini-2.5-Pro embeddings of human text and text from GPT-5 Chat with different prompt configurations. Points are colored by prompt type, with clusters labeled A through E representing distinct response patterns. Clusters were found and characterized through manual inspection of samples. Values between parentheses indicate mean pairwise euclidean distance to human text.
• Figure 3: Matrix of correlations among psychologists (P$_n$), the MindEval judge, and the average psychologist (Avg. P). Darker colors indicate stronger agreement. The values below the diagonal are Kendall-$\tau$ between annotators of the scores for every interaction. The ones above the diagonal are mean interaction-level pairwise system accuracy (MIPSA).
• Figure 4: MindEval performance comparison by criterion across different patient groups, interaction lengths, and prompt setups. Bars show mean scores, upward triangles indicate best-performing model scores, and downward triangles indicate worst-performing model scores for each criterion-setup combination. Refer to Table \ref{['tab:eval-guidelines-short']} for criteria descriptions.
• Figure 5: Patient profile example.