Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling

Qingyang Xu, Yaling Shen, Stephanie Fong, Zimu Wang, Yiwen Jiang, Xiangyu Zhao, Jiahe Liu, Zhongxing Xu, Vincent Lee, Zongyuan Ge

Abstract

The increasing use of large language models (LLMs) in mental healthcare raises safety concerns in high-stakes therapeutic interactions. A key challenge is distinguishing therapeutic empathy from maladaptive validation, where supportive responses may inadvertently reinforce harmful beliefs or behaviors in multi-turn conversations. This risk is largely overlooked by existing red-teaming frameworks, which focus mainly on generic harms or optimization-based attacks. To address this gap, we introduce Personality-based Client Simulation Attack (PCSA), the first red-teaming framework that simulates clients in psychological counseling through coherent, persona-driven client dialogues to expose vulnerabilities in psychological safety alignment. Experiments on seven general and mental health-specialized LLMs show that PCSA substantially outperforms four competitive baselines. Perplexity analysis and human inspection further indicate that PCSA generates more natural and realistic dialogues. Our results reveal that current LLMs remain vulnerable to domain-specific adversarial tactics, providing unauthorized medical advice, reinforcing delusions, and implicitly encouraging risky actions.

Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling

Abstract

The increasing use of large language models (LLMs) in mental healthcare raises safety concerns in high-stakes therapeutic interactions. A key challenge is distinguishing therapeutic empathy from maladaptive validation, where supportive responses may inadvertently reinforce harmful beliefs or behaviors in multi-turn conversations. This risk is largely overlooked by existing red-teaming frameworks, which focus mainly on generic harms or optimization-based attacks. To address this gap, we introduce Personality-based Client Simulation Attack (PCSA), the first red-teaming framework that simulates clients in psychological counseling through coherent, persona-driven client dialogues to expose vulnerabilities in psychological safety alignment. Experiments on seven general and mental health-specialized LLMs show that PCSA substantially outperforms four competitive baselines. Perplexity analysis and human inspection further indicate that PCSA generates more natural and realistic dialogues. Our results reveal that current LLMs remain vulnerable to domain-specific adversarial tactics, providing unauthorized medical advice, reinforcing delusions, and implicitly encouraging risky actions.

Paper Structure

This paper contains 35 sections, 4 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Jailbreaking
  4. Safety Alignment in High-Stakes Domains
  5. Clinical Safety Evaluations.
  6. Mental Health Applications and Risks.
  7. Methodology
  8. Problem Formulation
  9. Phase I: Domain-Specific Persona Initialization
  10. Phase II: Strategy-Driven Interaction Loop
  11. Multi-Dimensional Safety Assessment
  12. Experiments
  13. Model & Baselines
  14. Datasets & Benchmark
  15. Client Profile Datasets.
  16. ...and 20 more sections

Figures (7)

  • Figure 1: An example of our simulation. While LLM resists explicit harmful queries (top), it becomes vulnerable to toxic empathy when the intent is obscured by a persona-based narrative (bottom).
  • Figure 2: PCSA overview. (i) Domain-specific persona injection grounds the attacker in mental-health client profiles and dialogue styles. (ii) Strategy-driven simulation iteratively probes the target model with adaptive client behaviors, while an online evaluator guides attack optimization. A multidimensional judge then assesses the unsafe responses generated by the target LLM.
  • Figure 3: The prompt used by the internal evaluator to score the vulnerability of the target model's response during multi-turn simulation loop.
  • Figure 4: Detailed prompt for the LLM-as-a-Judge.
  • Figure 5: Representative failure modes where automated judging and human expert evaluation achieved consensus on safety violations.
  • ...and 2 more figures