Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums

TL;DR

This study assesses whether GPT-4 and GPT-3.5 can identify mental health factors in adolescent social media by performing open-information extraction on Reddit posts and comparing results to expert psychiatrists. It introduces a real Reddit dataset annotated by UK psychiatrists across six categories, plus two GPT-generated synthetic datasets, and evaluates both category-level and subcategory-level performance with positive/negative evidence annotations. The findings show GPT-4 achieving performance on par with human annotators on real data and substantially higher accuracy on synthetic data, while still making negation and factuality errors; synthetic data are easier to annotate but less diverse. The work suggests LLMs can enable scalable, low-cost mental-health factor extraction from adolescent content, with caveats about data diversity and annotation noise, and highlights synthetic data as a valuable, though not fully equivalent, resource for training tasks in healthcare contexts.

Abstract

Mental health in children and adolescents has been steadily deteriorating over the past few years. The recent advent of Large Language Models (LLMs) offers much hope for cost and time efficient scaling of monitoring and intervention, yet despite specifically prevalent issues such as school bullying and eating disorders, previous studies on have not investigated performance in this domain or for open information extraction where the set of answers is not predetermined. We create a new dataset of Reddit posts from adolescents aged 12-19 annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY and TREATMENT and compare expert labels to annotations from two top performing LLMs (GPT3.5 and GPT4). In addition, we create two synthetic datasets to assess whether LLMs perform better when annotating data as they generate it. We find GPT4 to be on par with human inter-annotator agreement and performance on synthetic data to be substantially higher, however we find the model still occasionally errs on issues of negation and factuality and higher performance on synthetic data is driven by greater complexity of real data rather than inherent advantage.

Figures (9)

• Figure 1: Distribution of subreddits
• Figure 2: Distribution of trauma subcategories (real data)
• Figure 3: Distribution of precarity subcategories (real data)
• Figure 4: Distribution of condition subcategories (real data) (subcategories with n > 2)
• Figure 5: Distribution of symptom subcategories (real data) (subcategories with n > 2)