Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech

TL;DR

This study tackles the problem of speech-based depression detection by leveraging multi-layer self-supervised representations. The authors introduce HAREN-CTC, a framework that fuses hierarchical SSL features through Hierarchical Adaptive Clustering and Cross-Modal Fusion, with a CTC objective to capture sparsely distributed depressive cues. The approach achieves state-of-the-art macro F1 scores on DAIC-WOZ (0.81) and MODMA (0.82) and demonstrates strong generalization in cross-validation, validated by comprehensive ablations. The work advances robust SDD by exploiting inter-layer interactions and weak temporal supervision, with practical implications for scalable, non-invasive mental health screening.

Abstract

Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments. However, it remains limited by the difficulty of extracting meaningful features and capturing sparse, heterogeneous depressive cues over time. Pretrained self-supervised learning (SSL) models such as WavLM provide rich, multi-layer speech representations, yet most existing SDD methods rely only on the final layer or search for a single best-performing one. These approaches often overfit to specific datasets and fail to leverage the full hierarchical structure needed to detect subtle and persistent depression signals. To address this challenge, we propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework, combined with Connectionist Temporal Classification loss to handle sparse temporal supervision. HAREN-CTC comprises two key modules: a Hierarchical Adaptive Clustering module that reorganizes SSL features into complementary embeddings, and a Cross-Modal Fusion module that models inter-layer dependencies through cross-attention. The CTC objective enables alignment-aware training, allowing the model to track irregular temporal patterns of depressive speech cues. We evaluate HAREN-CTC under both an upper-bound setting with standard data splits and a generalization setting using five-fold cross-validation. The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.

Figures (5)

• Figure 1: The architecture of HAREN-CTC (Hierarchical Acoustic Representation Encoding Network) for binary depression classification. It consists of: 1. Hierarchical Adaptive Clustering – groups transformer layer features to refine deep and shallow representations. 2. Cross-Modal Fusion – enhances interactions via bidirectional attention for depression-related features. 3. CTC-label Generation – produces weak labels using HuBERT, temporal pooling, and K-means clustering.
• Figure 2: Confusion matrix and ROC AUC Curve on the development set of the DAIC-WOZ dataset in performance upper-bound evaluation; (a) Confusion Matrix, (b) ROC AUC Curve.
• Figure 3: Comparison of the model's performance without HAREN on the DAIC-WOZ development set using representations from different hidden layers of the pretrained WavLM.
• Figure 4: Comparison of Performance Metrics for Different Numbers of Centroids (K = 5, 10, 15) on the DAIC-WOZ development set.
• Figure 5: Distribution of Acoustic Patterns in Depressed vs. Non-depressed group. Positive values indicate acoustic patterns more frequent in the depressed group, while negative values indicate patterns more frequent in the non-depressed group. Asterisks (*) indicate statistically significant differences (p < 0.05).