MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

TL;DR

This work tackles the absence of domain-specific pretrained language models for mental healthcare by training MentalBERT and MentalRoBERTa, initialized from BERT and RoBERTa checkpoints and continued pretraining on a large Reddit-based mental-health corpus. They systematically evaluate these models on depression, suicidal ideation, and other mental-disorder datasets, showing that domain-adaptive pretraining yields measurable improvements over general-domain and biomedical/clinical baselines. The models are publicly released, enabling researchers to leverage domain-specific representations for automated mental-health detection in online content. The study demonstrates the practical significance of targeted pretraining for public health monitoring and early intervention efforts, while acknowledging English-language limitation and ethical considerations.

Abstract

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.