Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval-Augmented Multimodal Depression Detection

Ruibo Hou, Shiyu Teng, Jiaqing Liu, Shurong Chai, Yinhao Li, Lanfen Lin, Yen-Wei Chen

TL;DR

This work tackles depression detection from multimodal signals by addressing limitations of static pre-training and cross-domain transfer. It introduces a Retrieval-Augmented Generation framework that dynamically retrieves emotionally relevant text and sentiment labels from an external dataset to generate an Emotion Prompt, forming a fourth modality. The Emotion Prompt is fused with text, audio, and video via cross-attention, improving both representation and interpretability. On the AVEC 2019 benchmark, the method achieves state-of-the-art performance (CCC ≈ 0.593, MAE ≈ 3.95), with ablation confirming the Emotion Prompt’s contribution; the approach reduces reliance on static weights and better adapts to task-specific emotional cues. This framework holds potential for broader affective computing applications where dynamic, context-aware emotion cues are beneficial.

Abstract

Multimodal deep learning has shown promise in depression detection by integrating text, audio, and video signals. Recent work leverages sentiment analysis to enhance emotional understanding, yet suffers from high computational cost, domain mismatch, and static knowledge limitations. To address these issues, we propose a novel Retrieval-Augmented Generation (RAG) framework. Given a depression-related text, our method retrieves semantically relevant emotional content from a sentiment dataset and uses a Large Language Model (LLM) to generate an Emotion Prompt as an auxiliary modality. This prompt enriches emotional representation and improves interpretability. Experiments on the AVEC 2019 dataset show our approach achieves state-of-the-art performance with CCC of 0.593 and MAE of 3.95, surpassing previous transfer learning and multi-task learning baselines.

Retrieval-Augmented Multimodal Depression Detection

TL;DR

This work tackles depression detection from multimodal signals by addressing limitations of static pre-training and cross-domain transfer. It introduces a Retrieval-Augmented Generation framework that dynamically retrieves emotionally relevant text and sentiment labels from an external dataset to generate an Emotion Prompt, forming a fourth modality. The Emotion Prompt is fused with text, audio, and video via cross-attention, improving both representation and interpretability. On the AVEC 2019 benchmark, the method achieves state-of-the-art performance (CCC ≈ 0.593, MAE ≈ 3.95), with ablation confirming the Emotion Prompt’s contribution; the approach reduces reliance on static weights and better adapts to task-specific emotional cues. This framework holds potential for broader affective computing applications where dynamic, context-aware emotion cues are beneficial.

Abstract

Multimodal deep learning has shown promise in depression detection by integrating text, audio, and video signals. Recent work leverages sentiment analysis to enhance emotional understanding, yet suffers from high computational cost, domain mismatch, and static knowledge limitations. To address these issues, we propose a novel Retrieval-Augmented Generation (RAG) framework. Given a depression-related text, our method retrieves semantically relevant emotional content from a sentiment dataset and uses a Large Language Model (LLM) to generate an Emotion Prompt as an auxiliary modality. This prompt enriches emotional representation and improves interpretability. Experiments on the AVEC 2019 dataset show our approach achieves state-of-the-art performance with CCC of 0.593 and MAE of 3.95, surpassing previous transfer learning and multi-task learning baselines.

Paper Structure

This paper contains 14 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. METHODOLOGY
  3. The Proposed Framework: An Overview
  4. Emotion Prompt via Retrieval-Augmented Generation
  5. EXPERIMENTAL SETTINGS
  6. Dataset
  7. AVEC 2019 Dataset
  8. CMU-MOSEI Dataset
  9. Implementation Details
  10. Evaluation Metrics
  11. EXPERIMENTAL RESULTS
  12. Performance Comparisons
  13. Ablation Study
  14. CONCLUSION

Figures (2)

  • Figure 1: Different multimodal depression detection frameworks. Unlike previous methods, our approach introduces the Emotion Prompt as a novel modality. By dynamically retrieving semantically similar text and sentiment labels from an external emotional dataset and integrating them with the original text, the Emotion Prompt serves as a fourth modality alongside text, audio, and video. This enables a more balanced multimodal fusion, significantly enhancing the representation of emotional cues and overall model performance.
  • Figure 2: (a) An overview of our proposed framework, where depression-related text and the Emotion Prompt are fed into the same pre-trained BERT model, while the audio and video encoders are trained from scratch. (b) The process of interacting with the LLM to generate the Emotion Prompt.