Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MHDash: An Online Platform for Benchmarking Mental Health-Aware AI Assistants

Yihe Zhang, Cheyenne N Mohawk, Kaiying Han, Vijay Srinivas Tida, Manyu Li, Xiali Hei

TL;DR

MHDash tackles the safety-critical challenge of evaluating mental health–aware AI in realistic, multi-turn settings by introducing a risk-aware benchmarking platform. It combines structured data collection, expert-in-the-loop annotation guided by $C$-SSRS, simulated multi-turn dialogues, and comprehensive risk-focused evaluation, including High-Risk Recall and Kendall’s $\tau$ for ordinal risk ranking. The work provides a 1,000-dialogue MHDialog dataset with an 8-intent taxonomy and a multi-layer pipeline for data, generation, modeling, and evaluation. Findings show that aggregate metrics can obscure dangerous failure modes, and that risk-aware analysis is essential for trustworthy, safety-aligned deployment of mental health AI systems.

Abstract

Large language models (LLMs) are increasingly applied in mental health support systems, where reliable recognition of high-risk states such as suicidal ideation and self-harm is safety-critical. However, existing evaluations primarily rely on aggregate performance metrics, which often obscure risk-specific failure modes and provide limited insight into model behavior in realistic, multi-turn interactions. We present MHDash, an open-source platform designed to support the development, evaluation, and auditing of AI systems for mental health applications. MHDash integrates data collection, structured annotation, multi-turn dialogue generation, and baseline evaluation into a unified pipeline. The platform supports annotations across multiple dimensions, including Concern Type, Risk Level, and Dialogue Intent, enabling fine-grained and risk-aware analysis. Our results reveal several key findings: (i) simple baselines and advanced LLM APIs exhibit comparable overall accuracy yet diverge significantly on high-risk cases; (ii) some LLMs maintain consistent ordinal severity ranking while failing absolute risk classification, whereas others achieve reasonable aggregate scores but suffer from high false negative rates on severe categories; and (iii) performance gaps are amplified in multi-turn dialogues, where risk signals emerge gradually. These observations demonstrate that conventional benchmarks are insufficient for safety-critical mental health settings. By releasing MHDash as an open platform, we aim to promote reproducible research, transparent evaluation, and safety-aligned development of AI systems for mental health support.

MHDash: An Online Platform for Benchmarking Mental Health-Aware AI Assistants

TL;DR

MHDash tackles the safety-critical challenge of evaluating mental health–aware AI in realistic, multi-turn settings by introducing a risk-aware benchmarking platform. It combines structured data collection, expert-in-the-loop annotation guided by -SSRS, simulated multi-turn dialogues, and comprehensive risk-focused evaluation, including High-Risk Recall and Kendall’s for ordinal risk ranking. The work provides a 1,000-dialogue MHDialog dataset with an 8-intent taxonomy and a multi-layer pipeline for data, generation, modeling, and evaluation. Findings show that aggregate metrics can obscure dangerous failure modes, and that risk-aware analysis is essential for trustworthy, safety-aligned deployment of mental health AI systems.

Abstract

Large language models (LLMs) are increasingly applied in mental health support systems, where reliable recognition of high-risk states such as suicidal ideation and self-harm is safety-critical. However, existing evaluations primarily rely on aggregate performance metrics, which often obscure risk-specific failure modes and provide limited insight into model behavior in realistic, multi-turn interactions. We present MHDash, an open-source platform designed to support the development, evaluation, and auditing of AI systems for mental health applications. MHDash integrates data collection, structured annotation, multi-turn dialogue generation, and baseline evaluation into a unified pipeline. The platform supports annotations across multiple dimensions, including Concern Type, Risk Level, and Dialogue Intent, enabling fine-grained and risk-aware analysis. Our results reveal several key findings: (i) simple baselines and advanced LLM APIs exhibit comparable overall accuracy yet diverge significantly on high-risk cases; (ii) some LLMs maintain consistent ordinal severity ranking while failing absolute risk classification, whereas others achieve reasonable aggregate scores but suffer from high false negative rates on severe categories; and (iii) performance gaps are amplified in multi-turn dialogues, where risk signals emerge gradually. These observations demonstrate that conventional benchmarks are insufficient for safety-critical mental health settings. By releasing MHDash as an open platform, we aim to promote reproducible research, transparent evaluation, and safety-aligned development of AI systems for mental health support.
Paper Structure (21 sections, 5 figures, 5 tables)

This paper contains 21 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Design
  4. Design Goals
  5. Design Overview
  6. Data Collection Layer
  7. Human-in-the-Loop Interaction Layer
  8. Conversation Generation Layer
  9. Modeling Layer
  10. Evaluation Layer
  11. Dataset
  12. Dialogue Intent Identification
  13. Dataset Construction
  14. Dataset Statistics
  15. Ethical Considerations
  16. ...and 6 more sections

Figures (5)

  • Figure 1: FNR Heatmap of 8 models to Concern Type and Risk Level.
  • Figure 2: Concern Type classification recall by category.
  • Figure 3: Risk Level classification recall by category.
  • Figure 4: Kendall Tau correlation for risk severity ranking across models.
  • Figure 5: Classification performance by dialogue intent, averaged across all models. Bars show accuracy for concern type (left) and risk level (right) classification, color-coded by performance level.