Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

TL;DR

This study introduces Automated Clinical AI Red Teaming, a domain-specific evaluation framework that simulates longitudinal psychotherapy with cognitive-affective patient agents to detect safety and quality issues in AI-based mental-health support. By applying the framework to AUD with six AI therapists and a 15-persona AUD cohort, the authors reveal critical iatrogenic risks such as in-session AI psychosis and failure to de-escalate crises, along with variability in therapeutic alliance and fidelity across models. The approach combines a comprehensive Qual ity of Care & Risk Ontology, a four-stage longitudinal evaluation cycle, and an interactive dashboard enabling diverse stakeholders to audit AI psychotherapy in a scalable, auditable manner. Findings highlight that safety cannot be achieved by prompt engineering alone, emphasize the need for simulation-based predeployment testing, and advocate for human-in-the-loop and policy safeguards before clinical deployment. The work provides a scalable blueprint for preclinical AI safety assessment in high-stakes conversational care and offers actionable insights for developers, clinicians, and regulators.

Abstract

Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and Character.AI) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.

Figures (8)

• Figure 1: The Four-Stage Cycle for Operationalizing the Ontology.
• Figure 2: High-Level Evaluation Framework Overview.
• Figure 3: Simulated Patient: Cognitive-Affective Pipeline.
• Figure 4: Longitudinal Analysis of the Working Alliance Inventory on the Dashboard.
• Figure 5: AI Psychosis Stage 1: Dehumanization.