HiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection

TL;DR

HiQuE introduces a Hierarchical Question Embedding network to detect depression from multimodal clinical interviews by explicitly modeling the hierarchy between primary and follow-up questions. The framework integrates a Question-Aware Module, Cross-Modal Attention, and a Depression Detection layer to fuse audio, visual, and text streams, with an 85-question embedding that maps 66 primary and 19 follow-up questions. Using DAIC-WOZ, with data augmentation to balance classes and RoBERTa for text, HiQuE achieves state-of-the-art macro F1, weighted F1, and G-mean scores and demonstrates generalization to unseen questions on E-DAIC-WOZ and MIT Interview. The approach provides interpretable multimodal cues through attention analyses and case studies, highlighting its potential for clinical deployment and extending to other speech-related tasks.

Abstract

The utilization of automated depression detection significantly enhances early intervention for individuals experiencing depression. Despite numerous proposals on automated depression detection using recorded clinical interview videos, limited attention has been paid to considering the hierarchical structure of the interview questions. In clinical interviews for diagnosing depression, clinicians use a structured questionnaire that includes routine baseline questions and follow-up questions to assess the interviewee's condition. This paper introduces HiQuE (Hierarchical Question Embedding network), a novel depression detection framework that leverages the hierarchical relationship between primary and follow-up questions in clinical interviews. HiQuE can effectively capture the importance of each question in diagnosing depression by learning mutual information across multiple modalities. We conduct extensive experiments on the widely-used clinical interview data, DAIC-WOZ, where our model outperforms other state-of-the-art multimodal depression detection models and emotion recognition models, showcasing its clinical utility in depression detection.

Figures (6)

• Figure 1: Previous research focused on learning the whole clinical interview sequences or their question-and-answer segments using a single modality. Our novel model, HiQuE, considers the hierarchy of questions incorporating multiple modalities to improve its effectiveness in depression detection. Numerical values in the boxes represent attention scores.
• Figure 2: An overall architecture of the HiQuE's multimodal depression detection process, where HIQ and Q-A indicate Hierarchical Question Embedded and Question-Aware, respectively.
• Figure 3: Hierarchical Question Embedding Process.
• Figure 4: Performance comparisons between unimodal and multimodal depression detection models. A, V, and T denote audio, visual, and text modality, respectively. The X-axis indicates the macro average precision, recall, and F1 score, while the Y-axis represents the corresponding scores.
• Figure 5: Distributions of attention scores across different modalities in each question, with the X-axis representing the Question ID from the first question ($Q_1$) to the last question ($Q_{85}$), and the Y-axis indicating the attention score.