Supporters and Skeptics: LLM-based Analysis of Engagement with Mental Health (Mis)Information Content on Video-sharing Platforms

TL;DR

This work tackles the spread and engagement of mental health misinformation in online video content by constructing the MentalMisinfo dataset (739 videos and 135,372 comments across YouTube Shorts and BitChute) with expert-annotated MHMisinfo labels based on three criteria. It demonstrates that few-shot in-context learning with multiple LLMs can outperform zero-shot and some gradient baselines in detecting MHMisinfo, enabling scalable labeling of unlabeled data to create a larger MentalMisinfo-Large corpus. The study then analyzes engagement linguistics (via LIWC and SAGE) and downstream signals of agreement and stigma, finding platform-specific patterns: YouTube comments responding to MHMisinfo more often exhibit risk, death language, and religion-oriented rhetoric with substantial though uneven agreement, whereas BitChute shows higher baseline agreement and different linguistic signals, likely reflecting platform populations. Finally, the paper discusses adaptive platform interventions, pathways to offline care, and ethical considerations, highlighting how technological and public-health approaches can mitigate online MH misinformation and its real-world harms.

Abstract

Over one in five adults in the US lives with a mental illness. In the face of a shortage of mental health professionals and offline resources, online short-form video content has grown to serve as a crucial conduit for disseminating mental health help and resources. However, the ease of content creation and access also contributes to the spread of misinformation, posing risks to accurate diagnosis and treatment. Detecting and understanding engagement with such content is crucial to mitigating their harmful effects on public health. We perform the first quantitative study of the phenomenon using YouTube Shorts and Bitchute as the sites of study. We contribute MentalMisinfo, a novel labeled mental health misinformation (MHMisinfo) dataset of 739 videos (639 from Youtube and 100 from Bitchute) and 135372 comments in total, using an expert-driven annotation schema. We first found that few-shot in-context learning with large language models (LLMs) are effective in detecting MHMisinfo videos. Next, we discover distinct and potentially alarming linguistic patterns in how audiences engage with MHMisinfo videos through commentary on both video-sharing platforms. Across the two platforms, comments could exacerbate prevailing stigma with some groups showing heightened susceptibility to and alignment with MHMisinfo. We discuss technical and public health-driven adaptive solutions to tackling the "epidemic" of mental health misinformation online.

Figures (1)

• Figure 1: Diagram summarizing our research design