Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues

David Gimeno-Gómez, Ana-Maria Bucur, Adrian Cosma, Carlos-David Martínez-Hinarejos, Paolo Rosso

TL;DR

This work tackles depression detection from user-generated videos by introducing a simple yet flexible multi-modal temporal model that leverages high-level non-verbal cues across time. By window-sampling diverse modalities (emotion-informed face embeddings, audio embeddings, body/hand landmarks, gaze and blinking) and using modality-conditioned, fractional positional embeddings fed into a transformer, the approach achieves state-of-the-art results on three benchmarks, including in-the-wild data. The authors demonstrate substantial performance gains, provide interpretability through Integrated Gradients to identify modality relevances over time, and show robustness to missing modalities. The method holds promise for preventive screening and early-warning applications on platforms with continuous video streams, while acknowledging ethical considerations and the non-clinical nature of the detection task.

Abstract

Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub.

Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues

TL;DR

This work tackles depression detection from user-generated videos by introducing a simple yet flexible multi-modal temporal model that leverages high-level non-verbal cues across time. By window-sampling diverse modalities (emotion-informed face embeddings, audio embeddings, body/hand landmarks, gaze and blinking) and using modality-conditioned, fractional positional embeddings fed into a transformer, the approach achieves state-of-the-art results on three benchmarks, including in-the-wild data. The authors demonstrate substantial performance gains, provide interpretability through Integrated Gradients to identify modality relevances over time, and show robustness to missing modalities. The method holds promise for preventive screening and early-warning applications on platforms with continuous video streams, while acknowledging ethical considerations and the non-clinical nature of the detection task.

Abstract

Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub.
Paper Structure (12 sections, 5 figures, 4 tables)

This paper contains 12 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Model Architecture
  5. Extracting and Encoding Non-Verbal Cues
  6. Experiments
  7. Datasets
  8. Evaluation
  9. Implementation Details
  10. Comparison Methods
  11. Results
  12. Conclusion

Figures (5)

  • Figure 1: The overall architecture of our proposed method. We extract high-level non-verbal cues using pretrained models, process them using a modality-specific encoder, condition the resulting embeddings with positional and modality embeddings, and process the sequence with a transformer encoder to perform the final classification.
  • Figure 2: Illustrative example of fractional positional embedding for temporally aligning video and audio sampling rates, similar to the work of Harzig et al. harzig2022fractional.
  • Figure 3: Distributions of video durations for each of our benchmarking datasets.
  • Figure 4: Presence distributions for each of our considered modalities in D-Vlog.
  • Figure 5: Attribution scores per each modality across frames obtained with Integrated Gradients sundararajan2017axiomatic on a selected window from a subject suffering from depression from D-Vlog. Higher values correspond to a strong attribution towards a positive prediction.