Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TRI-DEP: A Trimodal Comparative Study for Depression Detection Using Speech, Text, and EEG

Annisaa Fitri Nurfidausi, Eleonora Mancini, Paolo Torroni

TL;DR

This work tackles automatic depression detection using trimodal data (EEG, speech, and text) to overcome limitations of prior unimodal and bimodal studies. It introduces a unified, subject-level framework that systematically compares handcrafted features against pretrained embeddings, evaluates multiple encoders, and analyzes fusion strategies with careful attention to EEG. Unimodal results favor text and speech embeddings, while combining all three modalities via majority voting yields state-of-the-art performance (accuracy ≈ 88.6% and F1 ≈ 0.874). By providing reproducible preprocessing, feature extraction pipelines, and code, the paper establishes a solid foundation for robust, scalable multimodal depression detection research.

Abstract

Depression is a widespread mental health disorder, yet its automatic detection remains challenging. Prior work has explored unimodal and multimodal approaches, with multimodal systems showing promise by leveraging complementary signals. However, existing studies are limited in scope, lack systematic comparisons of features, and suffer from inconsistent evaluation protocols. We address these gaps by systematically exploring feature representations and modelling strategies across EEG, together with speech and text. We evaluate handcrafted features versus pre-trained embeddings, assess the effectiveness of different neural encoders, compare unimodal, bimodal, and trimodal configurations, and analyse fusion strategies with attention to the role of EEG. Consistent subject-independent splits are applied to ensure robust, reproducible benchmarking. Our results show that (i) the combination of EEG, speech and text modalities enhances multimodal detection, (ii) pretrained embeddings outperform handcrafted features, and (iii) carefully designed trimodal models achieve state-of-the-art performance. Our work lays the groundwork for future research in multimodal depression detection.

TRI-DEP: A Trimodal Comparative Study for Depression Detection Using Speech, Text, and EEG

TL;DR

This work tackles automatic depression detection using trimodal data (EEG, speech, and text) to overcome limitations of prior unimodal and bimodal studies. It introduces a unified, subject-level framework that systematically compares handcrafted features against pretrained embeddings, evaluates multiple encoders, and analyzes fusion strategies with careful attention to EEG. Unimodal results favor text and speech embeddings, while combining all three modalities via majority voting yields state-of-the-art performance (accuracy ≈ 88.6% and F1 ≈ 0.874). By providing reproducible preprocessing, feature extraction pipelines, and code, the paper establishes a solid foundation for robust, scalable multimodal depression detection research.

Abstract

Depression is a widespread mental health disorder, yet its automatic detection remains challenging. Prior work has explored unimodal and multimodal approaches, with multimodal systems showing promise by leveraging complementary signals. However, existing studies are limited in scope, lack systematic comparisons of features, and suffer from inconsistent evaluation protocols. We address these gaps by systematically exploring feature representations and modelling strategies across EEG, together with speech and text. We evaluate handcrafted features versus pre-trained embeddings, assess the effectiveness of different neural encoders, compare unimodal, bimodal, and trimodal configurations, and analyse fusion strategies with attention to the role of EEG. Consistent subject-independent splits are applied to ensure robust, reproducible benchmarking. Our results show that (i) the combination of EEG, speech and text modalities enhances multimodal detection, (ii) pretrained embeddings outperform handcrafted features, and (iii) carefully designed trimodal models achieve state-of-the-art performance. Our work lays the groundwork for future research in multimodal depression detection.
Paper Structure (13 sections, 1 figure, 4 tables)

This paper contains 13 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data
  4. Experimental Pipeline Design
  5. Data Preprocessing
  6. Feature Extraction
  7. Baselines
  8. Architectures
  9. Experimental Setup
  10. Results
  11. Conclusion
  12. Acknowledgments
  13. Compliance with Ethical Standards

Figures (1)

  • Figure 1: Experimental Framework for Multimodal Depression Detection