Depression detection in social media posts using transformer-based models and auxiliary features
Marios Kerasiotis, Loukas Ilias, Dimitris Askounis
TL;DR
The paper addresses depression severity detection in social media by integrating transformer-based contextual embeddings with metadata and linguistic markers to improve robustness and accuracy. It introduces a DistilBERT-based architecture that fuses hidden representations from the last four transformer layers with auxiliary features and processes them through an MLP classifier, complemented by Easy Data Augmentation to balance classes. Empirical results show a weighted F1-score improvement from 72.59% (baseline) to 84.15%, with strong performance against several baselines and thorough ablations on layer depth, model choice, and augmentation. The approach offers a scalable, efficient framework capable of adapting to different platforms and modalities, with implications for real-time mental health monitoring and epidemiological research.
Abstract
The detection of depression in social media posts is crucial due to the increasing prevalence of mental health issues. Traditional machine learning algorithms often fail to capture intricate textual patterns, limiting their effectiveness in identifying depression. Existing studies have explored various approaches to this problem but often fall short in terms of accuracy and robustness. To address these limitations, this research proposes a neural network architecture leveraging transformer-based models combined with metadata and linguistic markers. The study employs DistilBERT, extracting information from the last four layers of the transformer, applying learned weights, and averaging them to create a rich representation of the input text. This representation, augmented by metadata and linguistic markers, enhances the model's comprehension of each post. Dropout layers prevent overfitting, and a Multilayer Perceptron (MLP) is used for final classification. Data augmentation techniques, inspired by the Easy Data Augmentation (EDA) methods, are also employed to improve model performance. Using BERT, random insertion and substitution of phrases generate additional training data, focusing on balancing the dataset by augmenting underrepresented classes. The proposed model achieves weighted Precision, Recall, and F1-scores of 84.26%, 84.18%, and 84.15%, respectively. The augmentation techniques significantly enhance model performance, increasing the weighted F1-score from 72.59% to 84.15%.