Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Suicide Risk Assessment Using Multimodal Speech Features: A Study on the SW1 Challenge Dataset

Ambre Marie, Ilias Maoudj, Guillaume Dardenne, Gwenolé Quellec

TL;DR

This study tackles adolescent suicide risk assessment using a multimodal speech framework that combines WhisperX transcriptions, WavLM audio embeddings, Chinese RoBERTa text embeddings, and handcrafted acoustic features. It benchmarks three fusion strategies—early concatenation, modality-specific processing with attention, and weighted attention with mixup regularization—finding that weighted attention achieves the best generalization, though a notable gap remains between development and test performance. The work provides insights into how embedding representations and fusion mechanisms influence SR classification reliability within the MINI-KID framework, and it highlights generalization challenges inherent to speech-based risk assessment. Overall, the findings guide the design of more robust, multimodal SR systems and emphasize the need for richer datasets and labeling to translate development gains into real-world reliability.

Abstract

The 1st SpeechWellness Challenge conveys the need for speech-based suicide risk assessment in adolescents. This study investigates a multimodal approach for this challenge, integrating automatic transcription with WhisperX, linguistic embeddings from Chinese RoBERTa, and audio embeddings from WavLM. Additionally, handcrafted acoustic features -- including MFCCs, spectral contrast, and pitch-related statistics -- were incorporated. We explored three fusion strategies: early concatenation, modality-specific processing, and weighted attention with mixup regularization. Results show that weighted attention provided the best generalization, achieving 69% accuracy on the development set, though a performance gap between development and test sets highlights generalization challenges. Our findings, strictly tied to the MINI-KID framework, emphasize the importance of refining embedding representations and fusion mechanisms to enhance classification reliability.

Suicide Risk Assessment Using Multimodal Speech Features: A Study on the SW1 Challenge Dataset

TL;DR

This study tackles adolescent suicide risk assessment using a multimodal speech framework that combines WhisperX transcriptions, WavLM audio embeddings, Chinese RoBERTa text embeddings, and handcrafted acoustic features. It benchmarks three fusion strategies—early concatenation, modality-specific processing with attention, and weighted attention with mixup regularization—finding that weighted attention achieves the best generalization, though a notable gap remains between development and test performance. The work provides insights into how embedding representations and fusion mechanisms influence SR classification reliability within the MINI-KID framework, and it highlights generalization challenges inherent to speech-based risk assessment. Overall, the findings guide the design of more robust, multimodal SR systems and emphasize the need for richer datasets and labeling to translate development gains into real-world reliability.

Abstract

The 1st SpeechWellness Challenge conveys the need for speech-based suicide risk assessment in adolescents. This study investigates a multimodal approach for this challenge, integrating automatic transcription with WhisperX, linguistic embeddings from Chinese RoBERTa, and audio embeddings from WavLM. Additionally, handcrafted acoustic features -- including MFCCs, spectral contrast, and pitch-related statistics -- were incorporated. We explored three fusion strategies: early concatenation, modality-specific processing, and weighted attention with mixup regularization. Results show that weighted attention provided the best generalization, achieving 69% accuracy on the development set, though a performance gap between development and test sets highlights generalization challenges. Our findings, strictly tied to the MINI-KID framework, emphasize the importance of refining embedding representations and fusion mechanisms to enhance classification reliability.
Paper Structure (23 sections, 2 figures, 1 table)

This paper contains 23 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Materials and methods
  3. Dataset
  4. Transcription
  5. Embeddings
  6. Audio Embeddings
  7. Text Embeddings
  8. Acoustic Embeddings
  9. Fusion Strategies
  10. Early Concatenation
  11. Modality-Specific Processing with Attention
  12. Weighted Attention with Mixup Regularization
  13. Results
  14. Overview of Performance Metrics
  15. Embedding Representation
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Architecture of the Submission 3 classification pipeline.
  • Figure 2: Visualization of AuE for Submission 2. Left: Raw AuE extracted from WavLM Large. Right: Pre-logits AuE. Purple : label 0 (non-risk), Green : label 1 (at-risk).