MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Models

Abstract

Automated depression diagnosis aims to analyze multimodal information from interview videos to predict participants' depression scores. Previous studies often lack clear explanations of how these scores were determined, limiting their adoption in clinical practice. While the advent of LLMs provides a possible pathway for explainable depression diagnosis, current LLMs capable of processing multimodal data lack training on interview data, resulting in poor diagnostic performance when used directly. In this paper, we propose a novel multimodal large language model (MLlm-DR) that can understand multimodal information inputs and supports explainable depression diagnosis. MLlm-DR integrates a smaller LLMs and a lightweight query module (LQ-former). Specifically, the smaller LLMs is designed to generate depression scores and corresponding evaluation rationales. To enhance its logical reasoning for domain-specific tasks while maintaining practicality, we constructed a robust training dataset to fine-tune it. Meanwhile, the LQ-former captures depression-related features from speech and visual data, aiding the model's ability to process multimodal information, to achieve comprehensive depression diagnosis. Our approach achieves state-of-the-art results on two interview-based benchmark datasets, CMDC and E-DAIC-WOZ, demonstrating its effectiveness and superiority.

Figures (3)

• Figure 1: The Multimodal Large Language Model (MLlm-DR) is designed for explainable depression recognition. It leverages transcribed text, speech, and visual data from participants' interview videos to generate depression scores and corresponding evaluation rationales.
• Figure 2: The overall framework of the proposed MLlm-DR method. left shows the LQ-former pre-training part, which aims to extracts depression-related feature representations, comprehensible by LLMs, from visual and audio data. On the right is the LLMs fine-tuning part, where the learned feature representation is concatenated with text instruction embeddings as input to fine-tune the LLMs, which then output depression score and evaluation rationales.
• Figure 3: Case analysis of explainable depression recognition on the (a) CMDC and (b) E-DAIC-WOZ datasets. We present the inference results from three different models. "Dialogue content" refers to excerpts from the interview process, "label" represents the participant's true score, the highlighted orange section represents the model's predicted score, the highlighted yellow sections indicate the key parts of the dialogue content, and the red text indicates the key explanations in the evaluation rationale that are related to the corresponding scores.