Patterns vs. Patients: Evaluating LLMs against Mental Health Professionals on Personality Disorder Diagnosis through First-Person Narratives

TL;DR

This study probes whether state-of-the-art LLMs can diagnose personality disorders from raw first-person Polish narratives and how their reasoning compares to clinicians. By evaluating 16 models against 6 external experts on seven long autobiographical transcripts (3 BPD, 3 NPD, 1 healthy control), the authors demonstrate that Gemini Pro models substantially outperform humans in overall accuracy and show distinct biases in diagnosing NPD and in labeling healthy cases. While models provide confident, elaborative justifications focused on patterns, clinicians favor concise, patient-centered reasoning and exhibit cautious diagnostic language, highlighting safety concerns around AI certainty. The work suggests a collaborative human–AI framework to leverage diagnostic power while mitigating bias and reliability issues, and it emphasizes the need for broader, cross-cultural validation and multimodal data.

Abstract

Growing reliance on LLMs for psychiatric self-assessment raises questions about their ability to interpret qualitative patient narratives. We present the first direct comparison between state-of-the-art LLMs and mental health professionals in diagnosing Borderline (BPD) and Narcissistic (NPD) Personality Disorders utilizing Polish-language first-person autobiographical accounts. We show that the top-performing Gemini Pro models surpassed human professionals in overall diagnostic accuracy by 21.91 percentage points (65.48% vs. 43.57%). While both models and human experts excelled at identifying BPD (F1 = 83.4 & F1 = 80.0, respectively), models severely underdiagnosed NPD (F1 = 6.7 vs. 50.0), showing a reluctance toward the value-laden term "narcissism." Qualitatively, models provided confident, elaborate justifications focused on patterns and formal categories, while human experts remained concise and cautious, emphasizing the patient's sense of self and temporal experience. Our findings demonstrate that while LLMs are highly competent at interpreting complex first-person clinical data, they remain subject to critical reliability and bias issues.

Figures (5)

• Figure 1: Categorical and dimensional score (recalculated on a 0--1 scale) comparison between mental health professionals and models.
• Figure 2: Average model and mental health professionals' scores by true diagnosis.
• Figure 3: MDS and UMAP projections of the semantic embeddings of diagnosis justifications.
• Figure D.1: Diagnostic performance heatmap of human experts and models.
• Figure E.2: The 20 most representative lexical features for human experts and models by z-score.