LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment

TL;DR

LlaMADRS tackles automated depression severity assessment by prompting open-source LLMs to score MADRS items from transcribed clinical interviews. The approach leverages zero-shot prompts with Descriptive and Demonstrative cues and applies item-specific interview segmentation to CAMI data, achieving strong correlations with clinician ratings and near-human ICC on several items. Key contributions include a structured prompting strategy that yields high accuracy without training data, an empirical demonstration that segmented context improves performance, and a detailed error analysis revealing rater and patient characteristics as predictors. The findings suggest LLM-based depression assessment can enhance accessibility in resource-limited settings, while highlighting the need for multimodal data to address non-verbal cues and further clinical validation.

Abstract

This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment using the Montgomery-Asberg Depression Rating Scale (MADRS). We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews. Our approach, tested on 236 real-world interviews from the Context-Adaptive Multimodal Informatics (CAMI) dataset, demonstrates strong correlations with clinician assessments. The Qwen 2.5--72b model achieves near-human level agreement across most MADRS items, with Intraclass Correlation Coefficients (ICC) closely approaching those between human raters. We provide a comprehensive analysis of model performance across different MADRS items, highlighting strengths and current limitations. Our findings suggest that LLMs, with appropriate prompting, can serve as efficient tools for mental health assessment, potentially increasing accessibility in resource-limited settings. However, challenges remain, particularly in assessing symptoms that rely on non-verbal cues, underscoring the need for multimodal approaches in future work.

Figures (5)

• Figure 1: Overview of the LlaMADRS Framework. The left panel illustrates a traditional structured clinical interview between a patient and a clinician. The right panel demonstrates the automated depression assessment process using a large language model (Qwen 2.5 - 72b), including scoring of madrs items with explanations for each score.
• Figure 2: Montgomery-Åsberg Depression Rating Scale (madrs) Items. The scale includes ten items assessing different aspects of depression, providing a comprehensive evaluation of the patient's mental state.
• Figure 3: Mean absolute error (MAE) comparison between full transcript and item-specific context analysis across madrs items, with standard error bars (n=150). Item-specific processing demonstrates reduced error rates relative to full-transcript analysis (p < 0.01).
• Figure 4: Structured Prompt for MADRS Assessment. The prompt provides comprehensive guidance for analyzing psychiatric interview transcripts and assigning depression severity ratings. It includes the core components: task description, item definition, standardized questions, rating scale definitions, and required output format. This structure ensures consistent assessment across different raters and maintains compatibility with clinical standards.
• Figure 5: Representative Examples of LlaMADRS Assessment Performance. Comparison of two cases demonstrating the model's varying capability in MADRS item scoring. Example A shows accurate interpretation of reported sadness, matching the ground truth score of 2/6. Example B reveals a significant deviation from ground truth (4/6 vs 0/6), highlighting challenges in interpreting qualitative responses and temporal context for apparent sadness assessment.