Systematic Evaluation of Machine-Generated Reasoning and PHQ-9 Labeling for Depression Detection Using Large Language Models

TL;DR

This work tackles the reliability of large language models in detecting depression from social-media text by decomposing the task into 11 subtasks aligned with self-reference and PHQ-9 symptoms. It develops an end-to-end framework comprising expert annotation, prompting strategies, few-shot reasoning, and instruction tuning (SFT and DPO) to assess and improve LLM reasoning. Key findings show that explicit language cues boost subtask performance but joint depression diagnosis remains challenging, and that Direct Preference Optimization with sophisticated reasoning yields notable gains in overall accuracy. The study highlights the potential and limits of scalable, machine-generated reasoning for mental-health screening and argues for larger-scale data to validate and extend these results.

Abstract

Recent research leverages large language models (LLMs) for early mental health detection, such as depression, often optimized with machine-generated data. However, their detection may be subject to unknown weaknesses. Meanwhile, quality control has not been applied to these generated corpora besides limited human verifications. Our goal is to systematically evaluate LLM reasoning and reveal potential weaknesses. To this end, we first provide a systematic evaluation of the reasoning over machine-generated detection and interpretation. Then we use the models' reasoning abilities to explore mitigation strategies for enhanced performance. Specifically, we do the following: A. Design an LLM instruction strategy that allows for systematic analysis of the detection by breaking down the task into several subtasks. B. Design contrastive few-shot and chain-of-thought prompts by selecting typical positive and negative examples of detection reasoning. C. Perform human annotation for the subtasks identified in the first step and evaluate the performance. D. Identify human-preferred detection with desired logical reasoning from the few-shot generation and use them to explore different optimization strategies. We conducted extensive comparisons on the DepTweet dataset across the following subtasks: 1. identifying whether the speaker is describing their own depression; 2. accurately detecting the presence of PHQ-9 symptoms, and 3. finally, detecting depression. Human verification of statistical outliers shows that LLMs demonstrate greater accuracy in analyzing and detecting explicit language of depression as opposed to implicit expressions of depression. Two optimization methods are used for performance enhancement and reduction of the statistic bias: supervised fine-tuning (SFT) and direct preference optimization (DPO). Notably, the DPO approach achieves significant performance improvement.

Figures (4)

• Figure 1: An overview of the LLM-based detection and analysis framework.
• Figure 2: Evaluation of state-of-the-art Mental-LLMs with F1 scores (formatted as "LLM, F1%"). $TP$ denotes true positive, $FP$ false positive, $FN$ false negative, and $TN$ true negative. A higher $FP$ in the left column (Mentioned Depression (MD)) compared to the right (No Mention of Depression (NMD)) indicates that "the model has a tendency to consider the samples as depressed when these keywords are present".
• Figure 3: Evaluation of few-shot learning with LLMs w.r.t. different annotation groups (formatted as "LLM, F1%"). $TP$ denotes true positive, $FP$ false positive, $FN$ false negative, and $TN$ true negative.
• Figure 4: Distribution of samples over the $T_C$, $T_P$, and $T_W$ collections w.r.t. different annotation groups.