Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture of Experts for Recognizing Depression from Interview and Reading Tasks

Loukas Ilias, Dimitris Askounis

TL;DR

This work tackles automatic depression recognition from speech by integrating both read and spontaneous tasks through a multimodal fusion framework and a family of mixture-of-experts (MoE) models within a single end-to-end DNN. Audio from each task is turned into three-channel images and processed by shared AlexNet backbones, with fusion via BLOCK and subsequent inference through $mMoE$ variants, including sparse MoE and multilinear factorization CP and Tensor Ring MoEs. The Tensor Ring variant ($TR\u007Fm\u007FMoE$) delivers the best performance (approximately 87% accuracy and 86.66% F1 on the Androids corpus), outperforming single-modality baselines and other MoE variants. The results emphasize the value of input-conditional computation and parameter-efficient MoE architectures in small clinical datasets, while highlighting limitations related to dataset size and the need for future self-supervised and efficient fine-tuning methods.

Abstract

Depression is a mental disorder and can cause a variety of symptoms, including psychological, physical, and social. Speech has been proved an objective marker for the early recognition of depression. For this reason, many studies have been developed aiming to recognize depression through speech. However, existing methods rely on the usage of only the spontaneous speech neglecting information obtained via read speech, use transcripts which are often difficult to obtain (manual) or come with high word-error rates (automatic), and do not focus on input-conditional computation methods. To resolve these limitations, this is the first study in depression recognition task obtaining representations of both spontaneous and read speech, utilizing multimodal fusion methods, and employing Mixture of Experts (MoE) models in a single deep neural network. Specifically, we use audio files corresponding to both interview and reading tasks and convert each audio file into log-Mel spectrogram, delta, and delta-delta. Next, the image representations of the two tasks pass through shared AlexNet models. The outputs of the AlexNet models are given as input to a multimodal fusion method. The resulting vector is passed through a MoE module. In this study, we employ three variants of MoE, namely sparsely-gated MoE and multilinear MoE based on factorization. Findings suggest that our proposed approach yields an Accuracy and F1-score of 87.00% and 86.66% respectively on the Androids corpus.

Mixture of Experts for Recognizing Depression from Interview and Reading Tasks

TL;DR

This work tackles automatic depression recognition from speech by integrating both read and spontaneous tasks through a multimodal fusion framework and a family of mixture-of-experts (MoE) models within a single end-to-end DNN. Audio from each task is turned into three-channel images and processed by shared AlexNet backbones, with fusion via BLOCK and subsequent inference through variants, including sparse MoE and multilinear factorization CP and Tensor Ring MoEs. The Tensor Ring variant () delivers the best performance (approximately 87% accuracy and 86.66% F1 on the Androids corpus), outperforming single-modality baselines and other MoE variants. The results emphasize the value of input-conditional computation and parameter-efficient MoE architectures in small clinical datasets, while highlighting limitations related to dataset size and the need for future self-supervised and efficient fine-tuning methods.

Abstract

Depression is a mental disorder and can cause a variety of symptoms, including psychological, physical, and social. Speech has been proved an objective marker for the early recognition of depression. For this reason, many studies have been developed aiming to recognize depression through speech. However, existing methods rely on the usage of only the spontaneous speech neglecting information obtained via read speech, use transcripts which are often difficult to obtain (manual) or come with high word-error rates (automatic), and do not focus on input-conditional computation methods. To resolve these limitations, this is the first study in depression recognition task obtaining representations of both spontaneous and read speech, utilizing multimodal fusion methods, and employing Mixture of Experts (MoE) models in a single deep neural network. Specifically, we use audio files corresponding to both interview and reading tasks and convert each audio file into log-Mel spectrogram, delta, and delta-delta. Next, the image representations of the two tasks pass through shared AlexNet models. The outputs of the AlexNet models are given as input to a multimodal fusion method. The resulting vector is passed through a MoE module. In this study, we employ three variants of MoE, namely sparsely-gated MoE and multilinear MoE based on factorization. Findings suggest that our proposed approach yields an Accuracy and F1-score of 87.00% and 86.66% respectively on the Androids corpus.

Paper Structure

This paper contains 25 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Depression Recognition from Speech
  4. Traditional Machine Learning Algorithms
  5. Deep Neural Networks
  6. Mixture of Experts
  7. Dataset
  8. Methodology
  9. Speech Processing
  10. Reading Task
  11. Interview Task
  12. Deep Neural Network
  13. Fusion
  14. Mixture of Experts -- Output
  15. Sparse MoE
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Our proposed Methodology. Speech signals corresponding to interview and reading tasks are inputs to the deep neural network. These signals are transformed into log-Mel spectrogram, delta, and delta-delta. Next, they are passed through two pretrained shared AlexNet models. The outputs are then given as input to BLOCK (multimodal fusion method). The output vector is then passed through the Mixture of Experts module. Finally, an output layer with two units is used to differentiate healthy control from depression.
  • Figure 2: Test accuracy with respect to the number of experts