Leveraging Embedding Techniques in Multimodal Machine Learning for Mental Illness Assessment

TL;DR

This work addresses the need for objective, scalable mental health assessment by developing a comprehensive multimodal framework that integrates text, audio, and video signals from the E-DAIC dataset. It systematically evaluates embedding models across modalities, employs CNN-BiLSTM feature extractors, and explores data-, feature-, and decision-level fusion, including the novel integration of LLM predictions. The study extends to severity and multi-class tasks and demonstrates state-of-the-art performance for binary depression and PTSD detection, with robust generalization to severity estimation and co-morbidity classification. The findings highlight utterance-level chunking and decision-level fusion with LLM augmentation as particularly powerful, underscoring the potential for more accurate, accessible, and personalized mental health tools in clinical and at-home settings.

Abstract

The increasing global prevalence of mental disorders, such as depression and PTSD, requires objective and scalable diagnostic tools. Traditional clinical assessments often face limitations in accessibility, objectivity, and consistency. This paper investigates the potential of multimodal machine learning to address these challenges, leveraging the complementary information available in text, audio, and video data. Our approach involves a comprehensive analysis of various data preprocessing techniques, including novel chunking and utterance-based formatting strategies. We systematically evaluate a range of state-of-the-art embedding models for each modality and employ Convolutional Neural Networks (CNNs) and Bidirectional LSTM Networks (BiLSTMs) for feature extraction. We explore data-level, feature-level, and decision-level fusion techniques, including a novel integration of Large Language Model (LLM) predictions. We also investigate the impact of replacing Multilayer Perceptron classifiers with Support Vector Machines. We extend our analysis to severity prediction using PHQ-8 and PCL-C scores and multi-class classification (considering co-occurring conditions). Our results demonstrate that utterance-based chunking significantly improves performance, particularly for text and audio modalities. Decision-level fusion, incorporating LLM predictions, achieves the highest accuracy, with a balanced accuracy of 94.8% for depression and 96.2% for PTSD detection. The combination of CNN-BiLSTM architectures with utterance-level chunking, coupled with the integration of external LLM, provides a powerful and nuanced approach to the detection and assessment of mental health conditions. Our findings highlight the potential of MMML for developing more accurate, accessible, and personalized mental healthcare tools.

Figures (5)

• Figure 1: Interview data example as QA Format.
• Figure 2: Process Workflow
• Figure 3: Proposed Pipeline for Interview Analysis
• Figure 4: Fusion strategies in multi-modal data processing: (a) Data-Level Fusion - Multi-modal interview data (text, audio, video) undergoes embedding, fusion, and advanced feature extraction using MLPs, CNN, and BiLSTM before classification with SVM and MLPs. (b) Feature-Level Fusion - Extracted features from different modalities are combined for learning. (c) Decision-Level Fusion - Separate model outputs are integrated for final prediction.
• Figure 5: Comparison using MLP Classification layer vs SVM Classifier. For Binary and Multi-class classification the metric used BA, the MAE for severity. The heatmap illustrate the performance change when switching from an MLP to an SVM classifier.