A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies

TL;DR

This work systematically evaluates 11 state-of-the-art LLMs for PTSD severity estimation from open-ended narratives, using a clinical dataset of 1,437 participants with self-reported PCL-5 scores. It disentangles the influence of contextual knowledge (subscale definitions, elicitation context, distributions) and modeling strategies (model scale, prompting mode, reasoning effort, redistribution, and ensembling). Key findings show that rich contextual cues and calibrated post-processing substantially improve accuracy, with open-weight models plateauing around 70B while newer closed-weight models like GPT-5 surpass them; ensemble approaches that combine LLM outputs with supervised baselines achieve the best performance, even exceeding human raters on held-out data. The results offer practical guidance for deploying LLM-based mental health assessments, highlighting the value of prompt design, calibration, and model combination to achieve clinically meaningful estimates.

Abstract

Large language models (LLMs) are increasingly being used in a zero-shot fashion to assess mental health conditions, yet we have limited knowledge on what factors affect their accuracy. In this study, we utilize a clinical dataset of natural language narratives and self-reported PTSD severity scores from 1,437 individuals to comprehensively evaluate the performance of 11 state-of-the-art LLMs. To understand the factors affecting accuracy, we systematically varied (i) contextual knowledge like subscale definitions, distribution summary, and interview questions, and (ii) modeling strategies including zero-shot vs few shot, amount of reasoning effort, model sizes, structured subscales vs direct scalar prediction, output rescaling and nine ensemble methods. Our findings indicate that (a) LLMs are most accurate when provided with detailed construct definitions and context of the narrative; (b) increased reasoning effort leads to better estimation accuracy; (c) performance of open-weight models (Llama, Deepseek), plateau beyond 70B parameters while closed-weight (o3-mini, gpt-5) models improve with newer generations; and (d) best performance is achieved when ensembling a supervised model with the zero-shot LLMs. Taken together, the results suggest choice of contextual knowledge and modeling strategies is important for deploying LLMs to accurately assess mental health.

Figures (6)

• Figure 1: Effect of model size on PCL score estimation. The improvement rate of model performance degrades when scaling up from 70B to larger model variants.
• Figure 2: Model ensembles performance on PCL score estimation. Each marker shows an ensemble, positioned by Pearson correlation (x-axis, ↑ better) and MAE (y-axis, ↓ better). Circles denote direct score predictions; triangles denote subscale-based predictions. Marker slices indicate the constituent models (colors per legend), and marker size reflects ensemble size.
• Figure 3: Comparison of PTSD severity estimation accuracy across models and human raters. Bars show Pearson correlation ($r$) with ground-truth PCL scores on the held-out set of 187 interviews. GPT-5 and LLaMA-3.1-70B outperform both human raters and the supervised RoBERTa baseline kjell-et-al.
• Figure S4: Full layout of the PTSD subscale evaluation prompt. Colored boxes represent core components shared across all configurations. Gray boxes are optional plug-ins that can be included or excluded to control the amount of context provided.
• Figure S5: Further plug-in prompt components that were explored in this study. These elements provide additional information and are activated selectively. They can either be additional components (Additional Component) or alternative variants of a core component (Updated Component).